Publications related to Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Robust Collaborative Learning with Linear Gradient Overhead

Rachid Guerraoui, Nirupam Gupta, John Stephan, Sadegh Farhadkhani, Le Nguyen Hoang, Rafaël Benjamin Pinot

Collaborative learning algorithms, such as distributed SGD (or D-SGD), are prone to faulty machines that may deviate from their prescribed algorithm because of software or hardware bugs, poisoned data or malicious behaviors. While many solutions have been ...

PLMR2023

Incremental Learning in Diagonal Linear Networks

Raphaël Jean Berthier

Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient ...

Brookline2023

Functional data analysis with rough sample paths?

Victor Panaretos, Neda Mohammadi Jouzdani

Functional data are typically modeled as sample paths of smooth stochastic processes in order to mitigate the fact that they are often observed discretely and noisily, occasionally irregularly and sparsely. The smoothness assumption is imposed to allow for ...

TAYLOR & FRANCIS LTD2023

Universal and adaptive methods for robust stochastic optimization

Ali Kavis

Within the context of contemporary machine learning problems, efficiency of optimization process depends on the properties of the model and the nature of the data available, which poses a significant problem as the complexity of either increases ad infinit ...

EPFL2023

Byzantine Fault-Tolerance in Federated Local SGD Under 2f-Redundancy

Nirupam Gupta

In this article, we study the problem of Byzantine fault-tolerance in a federated optimization setting, where there is a group of agents communicating with a centralized coordinator. We allow up to

f

Byzantine-faulty agents, which may not follow a prescr ...

Piscataway2023

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Volkan Cevher, Alp Yurtsever, Maria-Luiza Vladarean

We propose a stochastic conditional gradient method (CGM) for minimizing convex finitesum objectives formed as a sum of smooth and non-smooth terms. Existing CGM variants for this template either suffer from slow convergence rates, or require carefully inc ...

2022

Countering Bias in Personalized Rankings From Data Engineering to Algorithm Development

Mirko Marras

This tutorial presents recent advances on the assessment and mitigation of data and algorithmic bias in personalized rankings. We first introduce fundamental concepts and definitions associated with bias issues, covering the state of the art and describing ...

IEEE COMPUTER SOC2021

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Rachid Guerraoui, El Mahdi El Mhamdi, Sébastien Louis Alexandre Rouault

Byzantine-resilient Stochastic Gradient Descent (SGD) aims at shielding model training from Byzantine faults, be they ill-labeled training datapoints, exploited software/hardware vulnerabilities, or malicious worker nodes in a distributed setting. Two rece ...

2021

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Lenka Zdeborová, Francesca Mignacco

In this paper we investigate how gradient-based algorithms such as gradient descent (GD), (multi-pass) stochastic GD, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best general ...

IOP PUBLISHING LTD2021

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Graph Chatbot

Chat with Graph Search

Robust Collaborative Learning with Linear Gradient Overhead

Incremental Learning in Diagonal Linear Networks

Functional data analysis with rough sample paths?

Universal and adaptive methods for robust stochastic optimization

Byzantine Fault-Tolerance in Federated Local SGD Under 2f-Redundancy

Towards Understanding Sharpness-Aware Minimization

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Countering Bias in Personalized Rankings From Data Engineering to Algorithm Development

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Universal and adaptive methods for robust stochastic optimization

Towards Understanding Sharpness-Aware Minimization

Byzantine Fault-Tolerance in Federated Local SGD Under 2f-Redundancy

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Incremental Learning in Diagonal Linear Networks

Functional data analysis with rough sample paths?

Robust Collaborative Learning with Linear Gradient Overhead

Countering Bias in Personalized Rankings From Data Engineering to Algorithm Development

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent