Publications related to Extrapolation for Large-batch Training in Deep Learning

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

This paper considers the Byzantine fault-tolerance problem in distributed stochastic gradient descent (D-SGD) method - a popular algorithm for distributed multi-agent machine learning. In this problem, each agent samples data points independently from a ce ...

IEEE COMPUTER SOC2021

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Nicolas Henri Bernard Flammarion, Scott William Pesme, Loucas Pillaud-Vivien

Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through i ...

2021

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Rachid Guerraoui, El Mahdi El Mhamdi, Sébastien Louis Alexandre Rouault

Byzantine-resilient Stochastic Gradient Descent (SGD) aims at shielding model training from Byzantine faults, be they ill-labeled training datapoints, exploited software/hardware vulnerabilities, or malicious worker nodes in a distributed setting. Two rece ...

2021

Second-Order Guarantees In Federated Learning

Ali H. Sayed, Stefan Vlaski, Elsa Rizk

Federated learning is a useful framework for centralized learning from distributed data under practical considerations of heterogeneity, asynchrony, and privacy. Federated architectures are frequently deployed in deep learning settings, which generally giv ...

IEEE2020

Deep Generative Models and Applications

Tatjana Chavdarova

Over the past few years, there have been fundamental breakthroughs in core problems in machine learning, largely driven by advances in deep neural networks. The amount of annotated data drastically increased and supervised deep discriminative models exceed ...

EPFL2020

Probabilistic Deep Learning on Spheres for Weather/Climate Applications

This work presents the application of a probabilistic approach to an already existing deep learning model for weather and climate prediction. Probabilistic deep learning allows to capture and address the uncertainties related to the data given as input and ...

2020

Square Attack: a query-efficient black-box adversarial attack via random search

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko, Francesco Croce

We propose the Square Attack, a new score-based black-box

l_2

and

l_\infty

adversarial attack that does not rely on local gradient information and thus is not affected by gradient masking. The Square Attack is based on a randomized search scheme where ...

2020

Reducing Noise in GAN Training with Variance Reduced Extragradient

François Fleuret, Tatjana Chavdarova

We study the effect of the stochastic gradient noise on the training of generative adversarial networks (GANs) and show that it can prevent the convergence of standard game optimization methods, while the batch version converges. We address this issue with ...

2019

Local SGD Converges Fast and Communicates Little

Sebastian Urban Stich

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large ne ...

2019

Geometry of adversarial robustness of deep networks: methods and applications

Seyed Mohsen Moosavi Dezfooli

We are witnessing a rise in the popularity of using artificial neural networks in many fields of science and technology. Deep neural networks in particular have shown impressive classification performance on a number of challenging benchmarks, generally in ...

EPFL2019

Extrapolation for Large-batch Training in Deep Learning

Graph Chatbot

Chat with Graph Search

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Second-Order Guarantees In Federated Learning

Deep Generative Models and Applications

Probabilistic Deep Learning on Spheres for Weather/Climate Applications

Square Attack: a query-efficient black-box adversarial attack via random search

Reducing Noise in GAN Training with Variance Reduced Extragradient

Local SGD Converges Fast and Communicates Little

Geometry of adversarial robustness of deep networks: methods and applications

Second-Order Guarantees In Federated Learning

Probabilistic Deep Learning on Spheres for Weather/Climate Applications

Local SGD Converges Fast and Communicates Little

Distributed Momentum for Byzantine-resilient Stochastic Gradient Descent

Reducing Noise in GAN Training with Variance Reduced Extragradient

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Deep Generative Models and Applications

Square Attack: a query-efficient black-box adversarial attack via random search

Geometry of adversarial robustness of deep networks: methods and applications

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination