Regularization (mathematics) | EPFL Graph Search

In mathematics, statistics, finance, computer science, particularly in machine learning and inverse problems, regularization is a process that changes the result answer to be "simpler". It is often used to obtain results for ill-posed problems or to prevent overfitting. Although regularization procedures can be divided in many ways, the following delineation is particularly helpful:

Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem. These terms could be priors, penalties, or constraints. Explicit regularization is commonly employed with ill-posed optimization problems. The regularization term, or penalty, imposes a cost on the optimization function to make the optimal solution unique.
Implicit regularization is all other forms of regularization. This includes, for example, early stopping, using a robust loss function, and discarding outliers. Implicit regularization is essentially ubiquitous in modern machine learning appr

Variational Methods For Continuous-Domain Inverse Problems: the Quest for the Sparsest Solution

Thomas Jean Debarre

The goal of this thesis is to study continuous-domain inverse problems for the reconstruction of sparse signals and to develop efficient algorithms to solve such problems computationally. The task is to recover a signal of interest as a continuous function from a finite number of measurements. This problem being severely ill-posed, we choose to favor sparse reconstructions. We achieve this by formulating an optimization problem with a regularization term involving the total-variation (TV) norm for measures. However, such problems often lead to nonunique solutions, some of which, contrary to expectations, may not be sparse. This requires particular care to assert that we reach a desired sparse solution.Our contributions are divided into three parts. In the first part, we propose exact discretization methods for large classes of TV-based problems with generic measurement operators for one-dimensional signals. Our methods are based on representer theorems that state that our problems have spline solutions. Our approach thus consists in performing an exact discretization of the problems in spline bases, and we propose algorithms which ensure that we reach a desired sparse solution. We then extend this approach to signals that are expressed as a sum of components with different characteristics. We either consider signals whose components are sparse in different bases or signals whose first component is sparse, and the other is smooth. In the second part, we consider more specific TV-based problems and focus on the identification of cases of uniqueness. Moreover, in cases of nonuniqueness, we provide a precise description of the solution set, and more specifically of the sparsest solutions. We then leverage this theoretical study to design efficient algorithms that reach such a solution. In this line, we consider the problem of interpolating one-dimensional data points with second-order TV regularization. We also study this same problem with an added Lipschitz constraint to favor stable solutions. Finally, we consider the problem of the recovery of periodic splines with low-frequency Fourier measurements, which we prove to always have a unique solution.In the third and final part, we apply our sparsity-based frameworks to various real-world problems. Our first application is a method for the fitting of sparse curves to contour data. Finally, we propose an image-reconstruction method for scanning transmission X-ray microscopy.

EPFL2022

3D Reconstruction of Optical Diffraction Tomography Based on a Neural Network Model

Morteza Hasani Shoreh

Optical tomography has been widely investigated for biomedical imaging applications. In recent years, it has been combined with digital holography and has been employed to produce high quality images of phase objects such as cells. In this Thesis, we look into some of the newest optical Diffraction Tomography (DT) based techniques to solve Three-Dimensional (3D) reconstruction problems and discuss and compare some of the leading ideas and papers. Then we propose a neural-network-based algorithm to solve this problem and apply it on both synthetic and biological samples. Conventional phase tomography with coherent light and off axis recording is performed. The Beam Propagation Method (BPM) is used to model scattering and each x-y plane is modeled by a layer of neurons in the BPM. The network's output (simulated data) is compared to the experimental measurements and the error is used for correcting the weights of the neurons (the refractive indices of the nodes) using standard error back-propagation techniques. The proposed algorithm is detailed and investigated. Then, we look into resolution-conserving regularization and discuss a method for selecting regularizing parameters. In addition, the local minima and phase unwrapping problems are discussed and ways of avoiding them are investigated. It is shown that the proposed learning tomography (LT) achieves better performance than other techniques such as, DT especially when insufficient number or incomplete set of measurements is available. We also explore the role of regularization in obtaining higher fidelity images without losing resolution. It is experimentally shown that due to overcoming multiple scattering, the LT reconstruction greatly outperforms the DT when the sample contains two or more layers of cells or beads. Then, reconstruction using intensity measurements is investigated. 3D reconstruction of a live cell during apoptosis is presented in a time-lapse format. At the end, we present a final comparison with leading papers and commercially available systems. It is shown that -compared to other existing algorithms- the results of the proposed method have better quality. In particular, parasitic granular structures and the missing cone artifact are improved. Overall, the perspectives of our approach are pretty rich for high-resolution tomographic imaging in a range of practical applications.

EPFL2018

Large Steps in Inverse Rendering of Geometry

Wenzel Alban Jakob, Baptiste Alexandre Marie Philippe Claude Nicolet

Inverse reconstruction from images is a central problem in many scientific and engineering disciplines. Recent progress on differentiable rendering has led to methods that can efficiently differentiate the full process of image formation with respect to millions of parameters to solve such problems via gradient-based optimization. At the same time, the availability of cheap derivatives does not necessarily make an inverse problem easy to solve. Mesh-based representations remain a particular source of irritation: an adverse gradient step involving vertex positions could turn parts of the mesh inside-out, introduce numerous local self-intersections, or lead to inadequate usage of the vertex budget due to distortion. These types of issues are often irrecoverable in the sense that subsequent optimization steps will further exacerbate them. In other words, the optimization lacks robustness due to an objective function with substantial non-convexity. Such robustness issues are commonly mitigated by imposing additional regularization, typically in the form of Laplacian energies that quantify and improve the smoothness of the current iterate. However, regularization introduces its own set of problems: solutions must now compromise between solving the problem and being smooth. Furthermore, gradient steps involving a Laplacian energy resemble Jacobi's iterative method for solving linear equations that is known for its exceptionally slow convergence. We propose a simple and practical alternative that casts differentiable rendering into the framework of preconditioned gradient descent. Our preconditioner biases gradient steps towards smooth solutions without requiring the final solution to be smooth. In contrast to Jacobi-style iteration, each gradient step propagates information among all variables, enabling convergence using fewer and larger steps. Our method is not restricted to meshes and can also accelerate the reconstruction of other representations, where smooth solutions are generally expected. We demonstrate its superior performance in the context of geometric optimization and texture reconstruction.

ASSOC COMPUTING MACHINERY2021