**Are you an EPFL student looking for a semester project?**

Work with us on data science and visualisation projects, and deploy your project as an app on top of GraphSearch.

Concept# Regularization (mathematics)

Summary

In mathematics, statistics, finance, computer science, particularly in machine learning and inverse problems, regularization is a process that changes the result answer to be "simpler". It is often used to obtain results for ill-posed problems or to prevent overfitting.
Although regularization procedures can be divided in many ways, the following delineation is particularly helpful:

- Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem. These terms could be priors, penalties, or constraints. Explicit regularization is commonly employed with ill-posed optimization problems. The regularization term, or penalty, imposes a cost on the optimization function to make the optimal solution unique.
- Implicit regularization is all other forms of regularization. This includes, for example, early stopping, using a robust loss function, and discarding outliers. Implicit regularization is essentially ubiquitous in modern machine learning appr

Official source

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications

Loading

Related people

Loading

Related units

Loading

Related concepts

Loading

Related courses

Loading

Related lectures

Loading

Related people (38)

Related publications (100)

Loading

Loading

Loading

Related units (16)

Related concepts (26)

Machine learning

Machine learning (ML) is an umbrella term for solving problems for which development of algorithms by human programmers would be cost-prohibitive, and instead the problems are solved by helping machin

Lasso (statistics)

In statistics and machine learning, lasso (least absolute shrinkage and selection operator; also Lasso or LASSO) is a regression analysis method that performs both variable selection and regularizat

Overfitting

In mathematical modeling, overfitting is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or pre

Related courses (26)

BIO-322: Introduction to machine learning for bioengineers

Students understand basic concepts and methods of machine learning. They can describe them in mathematical terms and can apply them to data using a high-level programming language (julia/python/R).

ME-371: Discretization methods in fluids

Ce cours présente une introduction aux méthodes d'approximation utilisées pour la simulation numérique en mécanique des fluides.
Les concepts fondamentaux sont présentés dans le cadre de la méthode des différences finies puis étendus à celles des éléments finis et spectraux.

MGT-418: Convex optimization

This course introduces the theory and application of modern convex optimization from an engineering perspective.

Related lectures (62)

Optical tomography has been widely investigated for biomedical imaging applications. In recent years, it has been combined with digital holography and has been employed to produce high quality images of phase objects such as cells. In this Thesis, we look into some of the newest optical Diffraction Tomography (DT) based techniques to solve Three-Dimensional (3D) reconstruction problems and discuss and compare some of the leading ideas and papers. Then we propose a neural-network-based algorithm to solve this problem and apply it on both synthetic and biological samples. Conventional phase tomography with coherent light and off axis recording is performed. The Beam Propagation Method (BPM) is used to model scattering and each x-y plane is modeled by a layer of neurons in the BPM. The network's output (simulated data) is compared to the experimental measurements and the error is used for correcting the weights of the neurons (the refractive indices of the nodes) using standard error back-propagation techniques. The proposed algorithm is detailed and investigated. Then, we look into resolution-conserving regularization and discuss a method for selecting regularizing parameters. In addition, the local minima and phase unwrapping problems are discussed and ways of avoiding them are investigated. It is shown that the proposed learning tomography (LT) achieves better performance than other techniques such as, DT especially when insufficient number or incomplete set of measurements is available. We also explore the role of regularization in obtaining higher fidelity images without losing resolution. It is experimentally shown that due to overcoming multiple scattering, the LT reconstruction greatly outperforms the DT when the sample contains two or more layers of cells or beads. Then, reconstruction using intensity measurements is investigated. 3D reconstruction of a live cell during apoptosis is presented in a time-lapse format. At the end, we present a final comparison with leading papers and commercially available systems. It is shown that -compared to other existing algorithms- the results of the proposed method have better quality. In particular, parasitic granular structures and the missing cone artifact are improved. Overall, the perspectives of our approach are pretty rich for high-resolution tomographic imaging in a range of practical applications.

Wenzel Alban Jakob, Baptiste Alexandre Marie Philippe Claude Nicolet

Inverse reconstruction from images is a central problem in many scientific and engineering disciplines. Recent progress on differentiable rendering has led to methods that can efficiently differentiate the full process of image formation with respect to millions of parameters to solve such problems via gradient-based optimization. At the same time, the availability of cheap derivatives does not necessarily make an inverse problem easy to solve. Mesh-based representations remain a particular source of irritation: an adverse gradient step involving vertex positions could turn parts of the mesh inside-out, introduce numerous local self-intersections, or lead to inadequate usage of the vertex budget due to distortion. These types of issues are often irrecoverable in the sense that subsequent optimization steps will further exacerbate them. In other words, the optimization lacks robustness due to an objective function with substantial non-convexity. Such robustness issues are commonly mitigated by imposing additional regularization, typically in the form of Laplacian energies that quantify and improve the smoothness of the current iterate. However, regularization introduces its own set of problems: solutions must now compromise between solving the problem and being smooth. Furthermore, gradient steps involving a Laplacian energy resemble Jacobi's iterative method for solving linear equations that is known for its exceptionally slow convergence. We propose a simple and practical alternative that casts differentiable rendering into the framework of preconditioned gradient descent. Our preconditioner biases gradient steps towards smooth solutions without requiring the final solution to be smooth. In contrast to Jacobi-style iteration, each gradient step propagates information among all variables, enabling convergence using fewer and larger steps. Our method is not restricted to meshes and can also accelerate the reconstruction of other representations, where smooth solutions are generally expected. We demonstrate its superior performance in the context of geometric optimization and texture reconstruction.

The goal of this thesis is to study continuous-domain inverse problems for the reconstruction of sparse signals and to develop efficient algorithms to solve such problems computationally. The task is to recover a signal of interest as a continuous function from a finite number of measurements. This problem being severely ill-posed, we choose to favor sparse reconstructions. We achieve this by formulating an optimization problem with a regularization term involving the total-variation (TV) norm for measures. However, such problems often lead to nonunique solutions, some of which, contrary to expectations, may not be sparse. This requires particular care to assert that we reach a desired sparse solution.Our contributions are divided into three parts. In the first part, we propose exact discretization methods for large classes of TV-based problems with generic measurement operators for one-dimensional signals. Our methods are based on representer theorems that state that our problems have spline solutions. Our approach thus consists in performing an exact discretization of the problems in spline bases, and we propose algorithms which ensure that we reach a desired sparse solution. We then extend this approach to signals that are expressed as a sum of components with different characteristics. We either consider signals whose components are sparse in different bases or signals whose first component is sparse, and the other is smooth. In the second part, we consider more specific TV-based problems and focus on the identification of cases of uniqueness. Moreover, in cases of nonuniqueness, we provide a precise description of the solution set, and more specifically of the sparsest solutions. We then leverage this theoretical study to design efficient algorithms that reach such a solution. In this line, we consider the problem of interpolating one-dimensional data points with second-order TV regularization. We also study this same problem with an added Lipschitz constraint to favor stable solutions. Finally, we consider the problem of the recovery of periodic splines with low-frequency Fourier measurements, which we prove to always have a unique solution.In the third and final part, we apply our sparsity-based frameworks to various real-world problems. Our first application is a method for the fitting of sparse curves to contour data. Finally, we propose an image-reconstruction method for scanning transmission X-ray microscopy.