Publications related to Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Phase Retrieval: From Computational Imaging to Machine Learning: A tutorial

Michaël Unser, Thanh-An Michel Pham, Jonathan Yuelin Dong

Phase retrieval consists in the recovery of a complex-valued signal from intensity-only measurements. As it pervades a broad variety of applications, many researchers have striven to develop phase-retrieval algorithms. Classical approaches involve techniqu ...

2023

Phase Reconstruction of Low-Energy Electron Holograms of Individual Proteins

Klaus Kern, Stephan Rauschenbach, Sven Alexander Szilagyi, Hannah Julia Ochner

Low-energy electron holography (LEEH) is one of the few techniques capable of imaging large and complex three-dimensional molecules, such as proteins, on the single molecule level at subnanometer resolution. During the imaging process, the structural infor ...

AMER CHEMICAL SOC2022

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Complexity analysis of stochastic gradient methods for PDE-constrained optimal control problems with uncertain parameters

Fabio Nobile, Sebastian Krumscheid, Matthieu Claude Martin

We consider the numerical approximation of an optimal control problem for an elliptic Partial Differential Equation (PDE) with random coefficients. Specifically, the control function is a deterministic, distributed forcing term that minimizes the expected ...

2021

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Nicolas Henri Bernard Flammarion, Scott William Pesme, Loucas Pillaud-Vivien

Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through i ...

2021

A Continuized View on Nesterov Acceleration

Nicolas Henri Bernard Flammarion, Raphaël Jean Berthier

We introduce the "continuized" Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuously mix following a linear ordinary differential equation and take gradien ...

2021

Large Steps in Inverse Rendering of Geometry

Wenzel Alban Jakob, Baptiste Alexandre Marie Philippe Claude Nicolet

Inverse reconstruction from images is a central problem in many scientific and engineering disciplines. Recent progress on differentiable rendering has led to methods that can efficiently differentiate the full process of image formation with respect to mi ...

ASSOC COMPUTING MACHINERY2021

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Florent Gérard Krzakala, Lenka Zdeborová, Francesca Mignacco

We analyze in a closed form the learning dynamics of stochastic gradient descent (SGD) for a single layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of ...

Curran Associates, Inc.2020

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Graph Chatbot

Chat with Graph Search

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Phase Retrieval: From Computational Imaging to Machine Learning: A tutorial

Phase Reconstruction of Low-Energy Electron Holograms of Individual Proteins

Masked Training of Neural Networks with Partial Gradients

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Complexity analysis of stochastic gradient methods for PDE-constrained optimal control problems with uncertain parameters

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

A Continuized View on Nesterov Acceleration

Large Steps in Inverse Rendering of Geometry

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Complexity analysis of stochastic gradient methods for PDE-constrained optimal control problems with uncertain parameters

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Large Steps in Inverse Rendering of Geometry

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

A Continuized View on Nesterov Acceleration

Phase Retrieval: From Computational Imaging to Machine Learning: A tutorial

Phase Reconstruction of Low-Energy Electron Holograms of Individual Proteins

Masked Training of Neural Networks with Partial Gradients

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification