Publications related to On the influence of momentum acceleration on online learning

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Brookline2024

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Volkan Cevher, Alp Yurtsever, Maria-Luiza Vladarean

We propose a stochastic conditional gradient method (CGM) for minimizing convex finitesum objectives formed as a sum of smooth and non-smooth terms. Existing CGM variants for this template either suffer from slow convergence rates, or require carefully inc ...

2022

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

Volkan Cevher, Kimon Antonakopoulos

Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem’s regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this regard, existin ...

2022

STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization

Volkan Cevher, Ali Kavis

In this work we investigate stochastic non-convex optimization problems wherethe objective is an expectation over smooth loss functions, and the goal is to find an approximate stationary point. The most popular approach to handling such problems is varianc ...

2021

Increasing Superstructure Optimization Capacity Through Self-Learning Surrogate Models

François Maréchal, Ivan Daniel Kantor, Julia Granacher

Simulation-based optimization models are widely applied to find optimal operating conditions of processes. Often, computational challenges arise from model complexity, making the generation of reliable design solutions difficult. We propose an algorithm fo ...

2021

Tomographic reconstruction of the runaway distribution function in TCV using multispectral synchrotron images

Basil Duval, Joan Decker, Umar Sheikh, Artur Perek, Gergely Papp

Synchrotron radiation observed in a quiescent Tokamak a Configuration Variable (TCV) runaway discharge is studied using filtered camera images targeting three distinct wavelength intervals. Through the tomographic simultaneous algebraic reconstruction tech ...

IOP PUBLISHING LTD2021

Computational Inverse Design of Surface-based Inflatables

Mark Pauly, Florin Isvoranu, Tian Chen

We present a computational inverse design method for a new class of surface-based inflatable structure. Our deployable structures are fabricated by fusing together two layers of inextensible sheet material along carefully selected curves. The fusing curves ...

ASSOC COMPUTING MACHINERY2021

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Rachid Guerraoui, El Mahdi El Mhamdi, Sébastien Louis Alexandre Rouault

Byzantine-resilient Stochastic Gradient Descent (SGD) aims at shielding model training from Byzantine faults, be they ill-labeled training datapoints, exploited software/hardware vulnerabilities, or malicious worker nodes in a distributed setting. Two rece ...

2021

Momentum space CFT correlators for Hamiltonian truncation

Matthew Thomas Walters

We consider Lorentzian CFT Wightman functions in momentum space. In particular, we derive a set of reference formulas for computing two- and three-point functions, restricting our attention to three-point functions where the middle operator (corresponding ...

2020

On the influence of momentum acceleration on online learning

Graph Chatbot

Chat with Graph Search

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

On the Generalization of Stochastic Gradient Descent with Momentum

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization

Increasing Superstructure Optimization Capacity Through Self-Learning Surrogate Models

Tomographic reconstruction of the runaway distribution function in TCV using multispectral synchrotron images

Computational Inverse Design of Surface-based Inflatables

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Momentum space CFT correlators for Hamiltonian truncation

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

On the Generalization of Stochastic Gradient Descent with Momentum

Tomographic reconstruction of the runaway distribution function in TCV using multispectral synchrotron images

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization

Computational Inverse Design of Surface-based Inflatables

Increasing Superstructure Optimization Capacity Through Self-Learning Surrogate Models

Momentum space CFT correlators for Hamiltonian truncation

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees