Publications related to The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Stochastic gradient descent (SGD) and randomized coordinate descent (RCD) are two of the workhorses for training modern automated decision systems. Intriguingly, convergence properties of these methods are not well-established as we move away from the spec ...

EPFL2021

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and—in asynchronous implementations—on the gradient staleness. Especially, it has been observed that the spe ...

2021

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Nikolaos Dimitriadis

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphol ...

IEEE2021

A first-order primal-dual method with adaptivity to local smoothness

Volkan Cevher, Maria-Luiza Vladarean

We consider the problem of finding a saddle point for the convex-concave objective

\min_x \max_y f(x) + \langle Ax, y\rangle - g^*(y)

, where

f

is a convex function with locally Lipschitz gradient and

g

is convex and possibly non-smooth. We propose an ...

2021

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Nicolas Henri Bernard Flammarion, Scott William Pesme, Loucas Pillaud-Vivien

Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through i ...

2021

Disparity Between Batches as a Signal for Early Stopping

Patrick Thiran, Mahsa Forouzesh

We propose a metric for evaluating the generalization ability of deep neural networks trained with mini-batch gradient descent. Our metric, called gradient disparity, is the l2 norm distance between the gradient vectors of two mini-batches drawn from the t ...

Springer2021

Gradient-based Methods for Deep Model Interpretability

Suraj Srinivas

In this dissertation, we propose gradient-based methods for characterizing model behaviour for the purposes of knowledge transfer and post-hoc model interpretation. Broadly, gradients capture the variation of some output feature of the model upon unit vari ...

EPFL2021

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Nirupam Gupta, Shuo Liu

This paper considers the Byzantine fault-tolerance problem in distributed stochastic gradient descent (D-SGD) method - a popular algorithm for distributed multi-agent machine learning. In this problem, each agent samples data points independently from a ce ...

IEEE COMPUTER SOC2021

Regret Minimization in Stochastic Non-Convex Learning via a Proximal-Gradient Approach

Volkan Cevher, Shaul Nadav Hallak

This paper develops a methodology for regret minimization with stochastic first-order oracle feedback in online, constrained, non-smooth, non-convex problems. In this setting, the minimization of external regret is beyond reach for first-order methods, and ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads

Martin Jaggi, Sebastian Urban Stich, Hossein Shokri Ghadikolaei

In distributed optimization, parameter updates from the gradient computing node devices have to be aggregated in every iteration on the orchestrating server. When these updates are sent over an arbitrary commodity network, bandwidth and latency can be limi ...

MICROTOME PUBLISHING2021

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

Graph Chatbot

Chat with Graph Search

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

A first-order primal-dual method with adaptivity to local smoothness

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Disparity Between Batches as a Signal for Early Stopping

Gradient-based Methods for Deep Model Interpretability

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Regret Minimization in Stochastic Non-Convex Learning via a Proximal-Gradient Approach

LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads

Adaptation in Stochastic Algorithms: From Nonsmooth Optimization to Min-Max Problems and Beyond

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

A first-order primal-dual method with adaptivity to local smoothness

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Regret Minimization in Stochastic Non-Convex Learning via a Proximal-Gradient Approach

LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Disparity Between Batches as a Signal for Early Stopping

Gradient-based Methods for Deep Model Interpretability