Publications related to Asynchronous updates for stochastic gradient descent

Simultaneous Perturbation Stochastic Approximation of the Quantum Fisher Information

Giuseppe Carleo, Julien Sebastian Gacon, Stefan Woerner

The Quantum Fisher Information matrix (QFIM) is a central metric in promising algorithms, such as Quantum Natural Gradient Descent and Variational Quantum Imaginary Time Evolution. Computing the full QFIM for a model with d parameters, however, is computat ...

VEREIN FORDERUNG OPEN ACCESS PUBLIZIERENS QUANTENWISSENSCHAF2021

Convergence of adaptive algorithms for constrained weakly convex optimization

Volkan Cevher, Ahmet Alacaoglu, Yurii Malitskyi

We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the

\tilde O(t^{-1/2})

rate of convergence for the squared norm of the gradient of Moreau envelope, ...

2021

Convergence without Convexity: Sampling, Optimization, and Games

Ya-Ping Hsieh

Many important problems in contemporary machine learning involve solving highly non- convex problems in sampling, optimization, or games. The absence of convexity poses significant challenges to convergence analysis of most training algorithms, and in some ...

EPFL2020

Second-order guarantees in centralized, federated and decentralized nonconvex optimization

Ali H. Sayed, Stefan Vlaski

Rapid advances in data collection and processing capabilities have allowed for the use of increasingly complex models that give rise to nonconvex optimization problems. These formulations, however, can be arbitrarily difficult to solve in general, in the s ...

INT PRESS BOSTON, INC2020

A new wake model and comparison of eight algorithms for layout optimization of wind farms in complex terrain

Fernando Porté Agel, Roberto Brogna

Layout optimization of wind farms constitutes an important and challenging task in complex terrain. This is especially due to the complex interactions of the boundary layer flows in complex terrain and wind turbine wakes, which renders wake modelling in co ...

ELSEVIER SCI LTD2020

Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms

Michel Bierlaire, Timothy Michael Hillel, Virginie Janine Camille Lurkin, Gael Lederrey

The emergence of Big Data has enabled new research perspectives in the discrete choice community. While the techniques to estimate Machine Learning models on a massive amount of data are well established, these have not yet been fully explored for the esti ...

2020

Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization

Ali H. Sayed, Kun Yuan

This paper introduces a new algorithm for consensus optimization in a multi-agent network, where all agents collaboratively find a minimizer for the sum of their private functions. All decentralized algorithms rely on communications between adjacent nodes. ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020

On the Influence of Bias-Correction on Distributed Stochastic Optimization

Ali H. Sayed, Bicheng Ying, Kun Yuan, Sulaiman A S A E Alghunaim

Various bias-correction methods such as EXTRA, gradient tracking methods, and exact diffusion have been proposed recently to solve distributed deterministic optimization problems. These methods employ constant step-sizes and converge linearly to the exact ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020

Global Convergence and Variance Reduction for a Class of Nonconvex-Nonconcave Minimax Problems

Negar Kiyavash

Nonconvex minimax problems appear frequently in emerging machine learning applications, such as generative adversarial networks and adversarial learning. Simple algorithms such as the gradient descent ascent (GDA) are the common practice for solving these ...

2020

A Catalyst Framework for Minimax Optimization

Negar Kiyavash

We introduce a generic \emph{two-loop} scheme for smooth minimax optimization with strongly-convex-concave objectives. Our approach applies the accelerated proximal point framework (or Catalyst) to the associated \emph{dual problem} and takes full advantag ...

2020

Asynchronous updates for stochastic gradient descent

Graph Chatbot

Chat with Graph Search

Simultaneous Perturbation Stochastic Approximation of the Quantum Fisher Information

Convergence of adaptive algorithms for constrained weakly convex optimization

Convergence without Convexity: Sampling, Optimization, and Games

Second-order guarantees in centralized, federated and decentralized nonconvex optimization

A new wake model and comparison of eight algorithms for layout optimization of wind farms in complex terrain

Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms

Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization

On the Influence of Bias-Correction on Distributed Stochastic Optimization

Global Convergence and Variance Reduction for a Class of Nonconvex-Nonconcave Minimax Problems

A Catalyst Framework for Minimax Optimization

Simultaneous Perturbation Stochastic Approximation of the Quantum Fisher Information

Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization

Convergence without Convexity: Sampling, Optimization, and Games

On the Influence of Bias-Correction on Distributed Stochastic Optimization

Estimation of discrete choice models with hybrid stochastic adaptive batch size algorithms

Second-order guarantees in centralized, federated and decentralized nonconvex optimization

A new wake model and comparison of eight algorithms for layout optimization of wind farms in complex terrain

A Catalyst Framework for Minimax Optimization

Global Convergence and Variance Reduction for a Class of Nonconvex-Nonconcave Minimax Problems

Convergence of adaptive algorithms for constrained weakly convex optimization