Publications related to Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

On the Generalization of Stochastic Gradient Descent with Momentum

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Brookline2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Byzantine Fault-Tolerance in Federated Local SGD Under 2f-Redundancy

Nirupam Gupta

In this article, we study the problem of Byzantine fault-tolerance in a federated optimization setting, where there is a group of agents communicating with a centralized coordinator. We allow up to

f

Byzantine-faulty agents, which may not follow a prescr ...

Piscataway2023

Deep Learning Generalization with Limited and Noisy Labels

Mahsa Forouzesh

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

EPFL2023

A review of the security vulnerabilities and countermeasures in the Internet of Things solutions: A bright future for the Blockchain

Jan Van Herle, Hossein Pourrahmani

The current advances in the Internet of Things (IoT) and the solutions being offered by this technology have accounted IoT among the top ten technologies that will transform the global economy by 2030. IoT is a state-of-the-art paradigm that has developed ...

2023

On the Double Descent of Random Features Models Trained with SGD

Volkan Cevher, Fanghui Liu

We study generalization properties of random features (RF) regression in high dimensions optimized by stochastic gradient descent (SGD) in under-/overparameterized regime. In this work, we derive precise non-asymptotic error bounds of RF regression under b ...

2022

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Volkan Cevher, Alp Yurtsever, Maria-Luiza Vladarean

We propose a stochastic conditional gradient method (CGM) for minimizing convex finitesum objectives formed as a sum of smooth and non-smooth terms. Existing CGM variants for this template either suffer from slow convergence rates, or require carefully inc ...

2022

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Patrick Thiran, Negar Kiyavash, Saber Salehkaleybar

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property with

1\le\alpha\le2

which holds in a wide range of applications in machine learning and signal processing. This conditio ...

NeurIPS2022

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Nirupam Gupta, Shuo Liu

This paper considers the Byzantine fault-tolerance problem in distributed stochastic gradient descent (D-SGD) method - a popular algorithm for distributed multi-agent machine learning. In this problem, each agent samples data points independently from a ce ...

IEEE COMPUTER SOC2021

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Graph Chatbot

Chat with Graph Search

On the Generalization of Stochastic Gradient Descent with Momentum

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Byzantine Fault-Tolerance in Federated Local SGD Under 2f-Redundancy

Deep Learning Generalization with Limited and Noisy Labels

A review of the security vulnerabilities and countermeasures in the Internet of Things solutions: A bright future for the Blockchain

On the Double Descent of Random Features Models Trained with SGD

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Towards Understanding Sharpness-Aware Minimization

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Deep Learning Generalization with Limited and Noisy Labels

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

On the Double Descent of Random Features Models Trained with SGD

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

A review of the security vulnerabilities and countermeasures in the Internet of Things solutions: A bright future for the Blockchain

Byzantine Fault-Tolerance in Federated Local SGD Under 2f-Redundancy

On the Generalization of Stochastic Gradient Descent with Momentum

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Towards Understanding Sharpness-Aware Minimization

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function