Publication
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
Related publications (32)
Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks
Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme
In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size and momentum parameter that allows u ...
2024Phase Retrieval: From Computational Imaging to Machine Learning: A tutorial
Michaël Unser, Thanh-An Michel Pham, Jonathan Yuelin Dong
Phase retrieval consists in the recovery of a complex-valued signal from intensity-only measurements. As it pervades a broad variety of applications, many researchers have striven to develop phase-retrieval algorithms. Classical approaches involve techniqu ...
2023Phase Reconstruction of Low-Energy Electron Holograms of Individual Proteins
Klaus Kern, Stephan Rauschenbach, Sven Alexander Szilagyi, Hannah Julia Ochner
Low-energy electron holography (LEEH) is one of the few techniques capable of imaging large and complex three-dimensional molecules, such as proteins, on the single molecule level at subnanometer resolution. During the imaging process, the structural infor ...
AMER CHEMICAL SOC2022Masked Training of Neural Networks with Partial Gradients
Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami
State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...
JMLR-JOURNAL MACHINE LEARNING RESEARCH2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks
Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro
Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...
2022Complexity analysis of stochastic gradient methods for PDE-constrained optimal control problems with uncertain parameters
Fabio Nobile, Sebastian Krumscheid, Matthieu Claude Martin
We consider the numerical approximation of an optimal control problem for an elliptic Partial Differential Equation (PDE) with random coefficients. Specifically, the control function is a deterministic, distributed forcing term that minimizes the expected ...
2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
Nicolas Henri Bernard Flammarion, Scott William Pesme, Loucas Pillaud-Vivien
Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through i ...
2021A Continuized View on Nesterov Acceleration
Nicolas Henri Bernard Flammarion, Raphaël Jean Berthier
We introduce the "continuized" Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuously mix following a linear ordinary differential equation and take gradien ...
2021Large Steps in Inverse Rendering of Geometry
Wenzel Alban Jakob, Baptiste Alexandre Marie Philippe Claude Nicolet
Inverse reconstruction from images is a central problem in many scientific and engineering disciplines. Recent progress on differentiable rendering has led to methods that can efficiently differentiate the full process of image formation with respect to mi ...
ASSOC COMPUTING MACHINERY2021