Publications related to On the influence of momentum acceleration on online learning

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Brookline2024

Universal and adaptive methods for robust stochastic optimization

Ali Kavis

Within the context of contemporary machine learning problems, efficiency of optimization process depends on the properties of the model and the nature of the data available, which poses a significant problem as the complexity of either increases ad infinit ...

EPFL2023

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

Volkan Cevher, Kimon Antonakopoulos

Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem’s regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this regard, existin ...

2022

Evidence of a coupled electron-phonon liquid in NbGe2

Mathieu François Padlewski, Philip Johannes Walter Moll, Matthias Carsten Putzke, Hao Yang

Whereas electron-phonon scattering relaxes the electron's momentum in metals, a perpetual exchange of momentum between phonons and electrons may conserve total momentum and lead to a coupled electron-phonon liquid. Such a phase of matter could be a platfor ...

NATURE PORTFOLIO2021

Tomographic reconstruction of the runaway distribution function in TCV using multispectral synchrotron images

Basil Duval, Joan Decker, Umar Sheikh, Artur Perek, Gergely Papp

Synchrotron radiation observed in a quiescent Tokamak a Configuration Variable (TCV) runaway discharge is studied using filtered camera images targeting three distinct wavelength intervals. Through the tomographic simultaneous algebraic reconstruction tech ...

IOP PUBLISHING LTD2021

Development of a novel two-phase flow solver for nuclear reactor analysis: algorithms, verification and implementation in OpenFOAM

Andreas Pautz, Carlo Fiorina, Stefan Radman

A porous medium-based representation of nuclear reactors and complex engineering systems more in general can significantly reduce simulation and modelling costs, while preserving a reasonable degree of accuracy via regime map-based correlations for modelli ...

2021

A first-order primal-dual method with adaptivity to local smoothness

Volkan Cevher, Maria-Luiza Vladarean

We consider the problem of finding a saddle point for the convex-concave objective

\min_x \max_y f(x) + \langle Ax, y\rangle - g^*(y)

, where

f

is a convex function with locally Lipschitz gradient and

g

is convex and possibly non-smooth. We propose an ...

2021

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and—in asynchronous implementations—on the gradient staleness. Especially, it has been observed that the spe ...

2021

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Ahmet Alacaoglu

Stochastic gradient descent (SGD) and randomized coordinate descent (RCD) are two of the workhorses for training modern automated decision systems. Intriguingly, convergence properties of these methods are not well-established as we move away from the spec ...

EPFL2021

On the influence of momentum acceleration on online learning

Graph Chatbot

Chat with Graph Search

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

On the Generalization of Stochastic Gradient Descent with Momentum

Universal and adaptive methods for robust stochastic optimization

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

Evidence of a coupled electron-phonon liquid in NbGe2

Tomographic reconstruction of the runaway distribution function in TCV using multispectral synchrotron images

Development of a novel two-phase flow solver for nuclear reactor analysis: algorithms, verification and implementation in OpenFOAM

A first-order primal-dual method with adaptivity to local smoothness

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Universal and adaptive methods for robust stochastic optimization

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Tomographic reconstruction of the runaway distribution function in TCV using multispectral synchrotron images

A first-order primal-dual method with adaptivity to local smoothness

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

On the Generalization of Stochastic Gradient Descent with Momentum

Evidence of a coupled electron-phonon liquid in NbGe2

Development of a novel two-phase flow solver for nuclear reactor analysis: algorithms, verification and implementation in OpenFOAM

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees