Publications related to Adaptive Gradient Descent without Descent

On The Convergence Of Stochastic Primal-Dual Hybrid Gradient

In this paper, we analyze the recently proposed stochastic primal-dual hybrid gradient (SPDHG) algorithm and provide new theoretical results. In particular, we prove almost sure convergence of the iterates to a solution with convexity and linear convergenc ...

SIAM PUBLICATIONS2022

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

Volkan Cevher, Kimon Antonakopoulos, Efstratios Panteleimon Skoulakis, Leello Tadesse Dadi, Ali Kavis

We propose an adaptive variance-reduction method, called AdaSpider, for minimization of L-smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider combines an AdaGrad-inspired [Duchi et al., 2011, McMahan & Streeter, 2010], but a fai ...

2022

Variational methods for finding periodic orbits in the incompressible Navier Stokes equations

Tobias Schneider, Jeremy Peter Parker

Unstable periodic orbits are believed to underpin the dynamics of turbulence, but by their nature are hard to find computationally. We present a family of methods to converge such unstable periodic orbits for the incompressible Navier-Stokes equations, bas ...

CAMBRIDGE UNIV PRESS2022

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

Volkan Cevher, Kimon Antonakopoulos

Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem’s regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this regard, existin ...

2022

A first-order primal-dual method with adaptivity to local smoothness

Volkan Cevher, Maria-Luiza Vladarean

We consider the problem of finding a saddle point for the convex-concave objective

\min_x \max_y f(x) + \langle Ax, y\rangle - g^*(y)

, where

f

is a convex function with locally Lipschitz gradient and

g

is convex and possibly non-smooth. We propose an ...

2021

Last iterate convergence of SGD for Least-Squares in the Interpolation regime

Nicolas Henri Bernard Flammarion, Aditya Vardhan Varre, Loucas Pillaud-Vivien

Motivated by the recent successes of neural networks that have the ability to fit the data perfectly \emph{and} generalize well, we study the noiseless model in the fundamental least-squares setup. We assume that an optimum predictor fits perfectly inputs ...

2021

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Ahmet Alacaoglu

Stochastic gradient descent (SGD) and randomized coordinate descent (RCD) are two of the workhorses for training modern automated decision systems. Intriguingly, convergence properties of these methods are not well-established as we move away from the spec ...

EPFL2021

A Newton Frank-Wolfe method for constrained self-concordant minimization

Volkan Cevher, Quoc Tran Dinh

We develop a new Newton Frank-Wolfe algorithm to solve a class of constrained self-concordant minimization problems using linear minimization oracles (LMO). Unlike L-smooth convex functions, where the Lipschitz continuity of the objective gradient holds gl ...

SPRINGER2021

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and—in asynchronous implementations—on the gradient staleness. Especially, it has been observed that the spe ...

2021

Computational Design of Weingarten Surfaces

Mark Pauly, Davide Pellis

In this paper we study Weingarten surfaces and explore their potential for fabrication-aware design in freeform architecture. Weingarten surfaces are characterized by a functional relation between their principal curvatures that implicitly defines approxim ...

ASSOC COMPUTING MACHINERY2021

Adaptive Gradient Descent without Descent

Graph Chatbot

Chat with Graph Search

On The Convergence Of Stochastic Primal-Dual Hybrid Gradient

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

Variational methods for finding periodic orbits in the incompressible Navier Stokes equations

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

A first-order primal-dual method with adaptivity to local smoothness

Last iterate convergence of SGD for Least-Squares in the Interpolation regime

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

A Newton Frank-Wolfe method for constrained self-concordant minimization

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Computational Design of Weingarten Surfaces

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

On The Convergence Of Stochastic Primal-Dual Hybrid Gradient

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

A Newton Frank-Wolfe method for constrained self-concordant minimization

Variational methods for finding periodic orbits in the incompressible Navier Stokes equations

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Last iterate convergence of SGD for Least-Squares in the Interpolation regime

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

A first-order primal-dual method with adaptivity to local smoothness

Computational Design of Weingarten Surfaces