Sebastian Urban Stich

Cette personne n’est plus à l’EPFL

À propos
Confidentialité
Mentions légales

Graph Chatbot

Publications associées (27)

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Martin Jaggi, Sebastian Urban Stich, Anastasiia Koloskova

Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-efficiency. In this paper we introduce a unified convergence analysis that cover ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2020

Dynamic Model Pruning with Feedback

Martin Jaggi, Luis Felipe Barba Flores, Sebastian Urban Stich, Tao Lin, Daniil Dmitriev

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that gener ...

2020

Ensemble Distillation for Robust Model Fusion in Federated Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the par ...

2020

Extrapolation for Large-batch Training in Deep Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batc ...

2020

Efficient Greedy Coordinate Descent for Composite Problems

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy, Anastasiia Koloskova

Coordinate descent with random coordinate selection is the current state of the art for many large scale optimization problems. However, greedy selection of the steepest coordinate on smooth problems can yield convergence rates independent of the dimension ...

2019

Local SGD Converges Fast and Communicates Little

Sebastian Urban Stich

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large ne ...

2019

Decentralized deep learning with arbitrary communication compression

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Anastasiia Koloskova

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches are limited by network bandwidth, we propose t ...

2019

Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication

Martin Jaggi, Sebastian Urban Stich, Anastasiia Koloskova

We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over n machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communi ...

2019

Don't Use Large Mini-Batches, Use Local SGD

Martin Jaggi, Sebastian Urban Stich, Tao Lin

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major ...

2019

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Martin Jaggi, Sebastian Urban Stich, Quentin Rebjock, Sai Praneeth Reddy Karimireddy

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD doe ...

PMLR2019

Page 2 sur 3