Publications related to Comparison of Gradient Estimation Methods for Real-time Optimization

Optimization Algorithms for Decentralized, Distributed and Collaborative Machine Learning

Distributed learning is the key for enabling training of modern large-scale machine learning models, through parallelising the learning process. Collaborative learning is essential for learning from privacy-sensitive data that is distributed across various ...

EPFL2024

Universal and adaptive methods for robust stochastic optimization

Ali Kavis

Within the context of contemporary machine learning problems, efficiency of optimization process depends on the properties of the model and the nature of the data available, which poses a significant problem as the complexity of either increases ad infinit ...

EPFL2023

Memory of Motion for Initializing Optimization in Robotics

Teguh Santoso Lembono

Many robotics problems are formulated as optimization problems. However, most optimization solvers in robotics are locally optimal and the performance depends a lot on the initial guess. For challenging problems, the solver will often get stuck at poor loc ...

EPFL2022

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Patrick Thiran, Negar Kiyavash, Saber Salehkaleybar

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property with

1\le\alpha\le2

which holds in a wide range of applications in machine learning and signal processing. This conditio ...

NeurIPS2022

Last iterate convergence of SGD for Least-Squares in the Interpolation regime

Nicolas Henri Bernard Flammarion, Aditya Vardhan Varre, Loucas Pillaud-Vivien

Motivated by the recent successes of neural networks that have the ability to fit the data perfectly \emph{and} generalize well, we study the noiseless model in the fundamental least-squares setup. We assume that an optimum predictor fits perfectly inputs ...

2021

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Ahmet Alacaoglu

Stochastic gradient descent (SGD) and randomized coordinate descent (RCD) are two of the workhorses for training modern automated decision systems. Intriguingly, convergence properties of these methods are not well-established as we move away from the spec ...

EPFL2021

Is there an analog of Nesterov acceleration for gradient-based MCMC?

Nicolas Henri Bernard Flammarion, Xiang Cheng

We formulate gradient-based Markov chain Monte Carlo (MCMC) sampling as optimization on the space of probability measures, with Kullback-Leibler (KL) divergence as the objective functional. We show that an under-damped form of the Langevin algorithm perfor ...

INT STATISTICAL INST2021

Modifier Adaptation as a Feedback Control Scheme

Dominique Bonvin, Alejandro Gabriel Marchetti, Sean Costello

As a real-time optimization technique, modifier adaptation (MA) has gained much significance in recent years. This is mainly due to the fact that MA can deal explicitly with structural plant-model mismatch and unknown disturbances. MA is an iterative techn ...

AMER CHEMICAL SOC2020

On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems

Volkan Cevher, Ali Kavis, Shaul Nadav Hallak

This paper analyzes the trajectories of stochastic gradient descent (SGD) to help understand the algorithm’s convergence properties in non-convex problems. We first show that the sequence of iterates generated by SGD remains bounded and converges with prob ...

2020

Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval

Florent Gérard Krzakala, Lenka Zdeborová

Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open problem. Here we fo ...

Curran Associates, Inc.2020

Comparison of Gradient Estimation Methods for Real-time Optimization

Graph Chatbot

Chat with Graph Search

Optimization Algorithms for Decentralized, Distributed and Collaborative Machine Learning

Universal and adaptive methods for robust stochastic optimization

Memory of Motion for Initializing Optimization in Robotics

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Last iterate convergence of SGD for Least-Squares in the Interpolation regime

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Is there an analog of Nesterov acceleration for gradient-based MCMC?

Modifier Adaptation as a Feedback Control Scheme

On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems

Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Is there an analog of Nesterov acceleration for gradient-based MCMC?

Modifier Adaptation as a Feedback Control Scheme

Universal and adaptive methods for robust stochastic optimization

Memory of Motion for Initializing Optimization in Robotics

On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems

Optimization Algorithms for Decentralized, Distributed and Collaborative Machine Learning

Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval

Last iterate convergence of SGD for Least-Squares in the Interpolation regime