Publications related to Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

Universal and adaptive methods for robust stochastic optimization

Within the context of contemporary machine learning problems, efficiency of optimization process depends on the properties of the model and the nature of the data available, which poses a significant problem as the complexity of either increases ad infinit ...

EPFL2023

Augmented Lagrangian Methods for Provable and Scalable Machine Learning

Mehmet Fatih Sahin

Non-convex constrained optimization problems have become a powerful framework for modeling a wide range of machine learning problems, with applications in k-means clustering, large- scale semidefinite programs (SDPs), and various other tasks. As the perfor ...

EPFL2023

The statistical complexity of early-stopped mirror descent

Tomas Vaskevicius, Varun Kanade

Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrain ...

Oxford2023

Stochastic distributed learning with gradient quantization and double-variance reduction

Sebastian Urban Stich, Konstantin Mishchenko

We consider distributed optimization over several devices, each sending incremental model updates to a central server. This setting is considered, for instance, in federated learning. Various schemes have been designed to compress the model updates in orde ...

TAYLOR & FRANCIS LTD2022

On The Convergence Of Stochastic Primal-Dual Hybrid Gradient

Volkan Cevher, Ahmet Alacaoglu

In this paper, we analyze the recently proposed stochastic primal-dual hybrid gradient (SPDHG) algorithm and provide new theoretical results. In particular, we prove almost sure convergence of the iterates to a solution with convexity and linear convergenc ...

SIAM PUBLICATIONS2022

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

Nicolas Boumal, Christopher Arnold Criscitiello

We describe the first gradient methods on Riemannian manifolds to achieve accelerated rates in the non-convex case. Under Lipschitz assumptions on the Riemannian gradient and Hessian of the cost function, these methods find approximate first-order critical ...

SPRINGER2022

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Ahmet Alacaoglu

Stochastic gradient descent (SGD) and randomized coordinate descent (RCD) are two of the workhorses for training modern automated decision systems. Intriguingly, convergence properties of these methods are not well-established as we move away from the spec ...

EPFL2021

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and—in asynchronous implementations—on the gradient staleness. Especially, it has been observed that the spe ...

2021

Convex optimization in sums of Banach spaces

Michaël Unser, Shayan Aziznejad

We characterize the solution of a broad class of convex optimization problems that address the reconstruction of a function from a finite number of linear measurements. The underlying hypothesis is that the solution is decomposable as a finite sum of compo ...

2021

A first-order primal-dual method with adaptivity to local smoothness

Volkan Cevher, Maria-Luiza Vladarean

We consider the problem of finding a saddle point for the convex-concave objective

\min_x \max_y f(x) + \langle Ax, y\rangle - g^*(y)

, where

f

is a convex function with locally Lipschitz gradient and

g

is convex and possibly non-smooth. We propose an ...

2021

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

Graph Chatbot

Chat with Graph Search

Universal and adaptive methods for robust stochastic optimization

Augmented Lagrangian Methods for Provable and Scalable Machine Learning

The statistical complexity of early-stopped mirror descent

Stochastic distributed learning with gradient quantization and double-variance reduction

On The Convergence Of Stochastic Primal-Dual Hybrid Gradient

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Convex optimization in sums of Banach spaces

A first-order primal-dual method with adaptivity to local smoothness

Augmented Lagrangian Methods for Provable and Scalable Machine Learning

Convex optimization in sums of Banach spaces

Universal and adaptive methods for robust stochastic optimization

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

On The Convergence Of Stochastic Primal-Dual Hybrid Gradient

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

A first-order primal-dual method with adaptivity to local smoothness

The statistical complexity of early-stopped mirror descent

Stochastic distributed learning with gradient quantization and double-variance reduction