Sparse Gradient Optimization and its Applications in Image Processing

Nikolaos Arvanitopoulos Darginis
EPFL, 2017
EPFL thesis

Abstract

Millions of digital images are captured by imaging devices on a daily basis. The way imaging devices operate follows an integral process from which the information of the original scene needs to be estimated. The estimation is done by inverting the integral process of the imaging device with the use of optimization techniques. This linear inverse problem, the inversion of the integral acquisition process, is at the heart of several image processing applications such as denoising, deblurring, inpainting, and super-resolution. We describe in detail the use of linear inverse problems in these applications. We review and compare several state-of-the-art optimization algorithms that invert this integral process. Linear inverse problems are usually very difficult to solve. Therefore, additional prior assumptions need to be introduced to successfully estimate the output signal. Several priors have been suggested in the research literature, with the Total Variation (TV) being one of the most prominent. In this thesis, we review another prior, the l0 pseudo-norm over the gradient domain. This prior allows full control over how many non-zero gradients are retained to approximate prominent structures of the image. We show the superiority of the l0 gradient prior over the TV prior in recovering genuinely piece-wise constant signals. The l0 gradient prior has shown to produce state-of-the-art results in edge-preserving image smoothing. Moreover, this general prior can be applied to several other applications, such as edge extraction, clip-art JPEG artifact removal, non-photorealistic image rendering, detail magnification, and tone mapping. We review and evaluate several state-of-the-art algorithms that solve the optimization problem based on the l0 gradient prior. Subsequently we apply the l0 gradient prior to two applications where we show superior results as compared to the current state-of-the-art. The first application is that of single-image reflection removal. Existing solutions to this problem have shown limited success because of the highly ill-posed nature of the problem. We show that the standard l0 gradient prior with a modified data-fidelity term based on the Laplacian operator is able to sufficiently remove unwanted reflections from images in many realistic scenarios. We conduct extensive experiments and show that our method outperforms the state-of-the-art. In the second application of haze removal from visible-NIR image pairs we propose a novel optimization framework, where the prior term penalizes the number of non-zero gradients of the difference between the output and the NIR image. Due to the longer wavelengths of NIR, an image taken in the NIR spectrum suffers significantly less from haze artifacts. Using this prior term, we are able to transfer details from the haze-free NIR image to the final result. We show that our formulation provides state-of-the-art results compared to haze removal methods that use a single image and also to those that are based on visible-NIR image pairs.

Official source

https://infoscience.epfl.ch/record/230439?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

Nikolaos Arvanitopoulos Darginis
EPFL, 2017
EPFL thesis

Abstract

Official source

https://infoscience.epfl.ch/record/230439?ln=en

About this result

Related concepts (30)

Related concepts

Related publications (34)

Related publications

Related publications (34)

Trust-region methods based on radial basis functions with application to biomedical imaging

Rodrigue Oeuvray

We have developed a new derivative-free algorithm based on Radial Basis Functions (RBFs). Derivative-free optimization is an active field of research and several algorithms have been proposed recently. Problems of this nature in the industrial setting are quite frequent. The reason is that in a number of applications the optimization process contains simulation packages which are treated as black boxes. The development of our own algorithm was originally motivated by an application in biomedical imaging: the medical image registration problem. The particular characteristics of this problem have incited us to develop a new optimization algorithm based on trust-region methods. However it has been designed to be generic and to be applied to a wide range of problems. The main originality of our approach is the use of RBFs to build the models. In particular we have adapted the existing theory based on quadratic models to our own models and developed new procedures especially designed for models based on RBFs. We have tested our algorithm called BOOSTERS against state-of-the-art methods (UOBYQA, NEWUOA, DFO). On the medical image registration problem, BOOSTERS appears to be the method of choice. The tests on problems from the CUTEr collection show that BOOSTERS is comparable to, but not better than other methods on small problems (size 2-20). It is performing very well for medium size problems (20-80). Moreover, it is able to solve problems of dimension 200, which is considered very large in derivative-free optimization. We have also developed a new class of algorithms combining the robustness of derivative-free algorithms with the faster rate of convergence characterizing Newtonlike-methods. In fact, they define a new class of algorithms lying between derivative-free optimization and quasi-Newton methods. These algorithms are built on the skeleton of our derivative-free algorithm but they can incorporate the gradient when it is available. They can be interpreted as a way of doping derivative-free algorithms with derivatives. If the derivatives are available at each iteration, then our method can be seen as an alternative to quasi-Newton methods. At the opposite, if the derivatives are never evaluated, then the algorithm is totally similar to BOOSTERS. It is a very interesting alternative to existing methods for problems whose objective function is expensive to evaluate and when the derivatives are not available. In this situation, the gradient can be approximated by finite differences and its costs corresponds to n additional function evaluations assuming that Rn is the domain of definition of the objective function. We have compared our method with CFSQP and BTRA, two gradient-based algorithms, and the results show that our doped method performs best. We have also a theoretical analysis of the medical image registration problem based on maximization of mutual information. Most of the current research in this field is concentrated on registration based on nonlinear image transformation. However, little attention has been paid to the theoretical properties of the optimization problem. In our analysis, we focus on the continuity and the differentiability of the objective function. We show in particular that performing a registration without extension of the reference image may lead to discontinuities in the objective function. But we demonstrate that, under some mild assumptions, the function is differentiable almost everywhere. Our analysis is important from an optimization point of view and conditions the choice of a solver. The usual practice is to use generic optimization packages without worrying about the differentiability of the objective function. But the use of gradient-based methods when the objective function is not differentiable may result in poor performance or even in absence of convergence. One of our objectives with this analysis is also that practitioners become aware of these problems and to propose them new algorithms having a potential interest for their applications.

EPFL2005

In the past few years, fusing NIR and color images has been explored in general computational photography and computer vision tasks, where traditionally only color images are used. The additional information provided by the differences of light and scene reflections in the visible and NIR bands of the electromagnetic spectrum is used in several applications such as image denoising, image dehazing, shadow detection and removal, and high dynamic-range imaging. In this thesis, we study a system that simultaneously captures color and NIR images on a single silicon sensor. Such a camera could be manufactured with minor changes in the hardware of consumer color cameras and it could be integrated inside small devices such as cell phones. We address two main challenges in color and NIR acquisition. First, we study the spatial and spectral sampling of the scene, which is inevitable in single-sensor acquisition of multiple spectral channels. We then focus on chromatic aberration distortions. Similarly to color imaging, we use a color filter array (CFA) to sample the scene in visible and NIR bands. We address two main challenges regarding the CFA: (1) designing the CFA, and (2) developing a demosaicing algorithm to reconstruct full-resolution images from the subsampled measurements. We consider a general CFA with filters that transmit different mixtures of color and NIR channels. We develop a framework that, by exploiting the spatial and spectral correlations of color and NIR images, computes the transmittance of each filter and the demosaicing matrix. Our optimized CFA and demosaicing outperform other solutions developed for single-sensor color and NIR acquisition. We also investigate a CFA that is formed by one blue, one green, one red, and one NIR-pass filter. We call it the RGBN CFA and assume that, similarly to color cameras, it uses dye filters that do not have sharp cut-offs. Hence, color and NIR radiations leak into NIR and color filters, respectively. We devise an algorithm that reconstructs full-resolution images from mixed and subsampled sensor measurements. The RGBN CFA and our reconstruction algorithm perform as well as or even better than other single-sensor acquisition techniques that use more complicated hardware components. The problem of chromatic aberration is caused by deficiencies of optical elements. A simple lens converges light rays with different wavelengths at different distances from the lens. Hence, if the color image is in focus and sharp on the sensor plane, the NIR image captured with the same focus settings is out of focus and blurred. We propose an algorithm that retrieves the lost details in NIR using the gradients of the sharp color image. As the high-frequency details of color and NIR images are not strongly correlated in all image patches, our method locally adapts the contribution of color gradients in deblurring. To achieve this, we develop a multiscale scheme that iterates between deblurring NIR and estimating the correlation between color and NIR high-frequency components. Our algorithm outperforms both blind and guided deblurring approaches. We also design a method that estimates a dense blur-kernel map when the severity of chromatic aberration changes as the depths of objects vary across the image. Our method performs better than the competing methods both in estimating the blur-kernel map and in deblurring.

EPFL2015

Shapes from Pixels

Mitra Fatemi

In today's digital world, sampling is at the heart of any signal acquisition device. Imaging devices are ubiquitous examples that capture two-dimensional visual signals and store them as the pixels of discrete images. The main concern is whether and how the pixels provide an exact or at least a fair representation of the original visual signal in the continuous domain. This motivates the design of exact reconstruction or approximation techniques for a target class of images. Such techniques benefit different imaging tasks such as super-resolution, deblurring and compression. This thesis focuses on the reconstruction of visual signals representing a shape over a background, from their samples. Shape images have only two intensity values. However, the filtering effect caused by the sampling kernel of imaging devices smooths out the sharp transitions in the image and results in samples with varied intensity levels. To trace back the shape boundaries, we need strategies to reconstruct the original bilevel image. But, abrupt intensity changes along the shape boundaries as well as diverse shape geometries make reconstruction of this class of signals very challenging. Curvelets and contourlets have been proved as efficient multiresolution representations for the class of shape images. This motivates the approximation of shape images in the aforementioned domains. In the first part of this thesis, we study generalized sampling and infinite-dimensional compressed sensing to approximate a signal in a domain that is known to provide a sparse or efficient representation for the signal, given its samples in a different domain. We show that the generalized sampling, due to its linearity, is incapable of generating good approximation of shape images from a limited number of samples. The infinite-dimensional compressed sensing is a more promising approach. However, the concept of random sampling in this scheme does not apply to the shape reconstruction problem. Next, we propose a sampling scheme for shape images with finite rate of innovation (FRI). More specifically, we model the shape boundaries as a subset of an algebraic curve with an implicit bivariate polynomial. We show that the image parameters are solutions of a set of linear equations with the coefficients being the image moments. We then replace conventional moments with more stable generalized moments that are adjusted to the given sampling kernel. This leads to successful reconstruction of shapes with moderate complexities from samples generated with realistic sampling kernels and in the presence of moderate noise levels. Our next contribution is a scheme for recovering shapes with smooth boundaries from a set of samples. The reconstructed image is constrained to regenerate the same samples (consistency) as well as forming a bilevel image. We initially formulate the problem by minimizing the shape perimeter over the set of consistent shapes. Next, we relax the non-convex shape constraint to transform the problem into minimizing the total variation over consistent non-negative-valued images. We introduce a requirement -called reducibility- that guarantees equivalence between the two problems. We illustrate that the reducibility effectively sets a requirement on the minimum sampling density. Finally, we study a relevant problem in the Boolean algebra: the Boolean compressed sensing. The problem is about recovering a sparse Boolean vector from a few collective binary tests. We study a formulation of this problem as a binary linear program, which is NP hard. To overcome the computational burden, we can relax the binary constraint on the variables and apply a rounding to the solution. We replace the rounding procedure with a randomized algorithm. We show that the proposed algorithm considerably improves the success rate with only a slight increase in the computational cost.

EPFL2016