Publications de Sai Praneeth Reddy Karimireddy

Learning from History for Byzantine Robust Optimization

Martin Jaggi, Sai Praneeth Reddy Karimireddy, Lie He

Byzantine robustness has received significant attention recently given its importance for distributed and federated learning. In spite of this, we identify severe flaws in existing algorithms even when the data across the participants is identically distri ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Martin Jaggi, Tao Lin, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization ch ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

Optimization methods for collaborative learning

Sai Praneeth Reddy Karimireddy

A traditional machine learning pipeline involves collecting massive amounts of data centrally on a server and training models to fit the data. However, increasing concerns about the privacy and security of user's data, combined with the sheer growth in the ...

EPFL2021

Weight Erosion: An Update Aggregation Scheme for Personalized Collaborative Machine Learning

Martin Jaggi, Mary-Anne Hartley, Sai Praneeth Reddy Karimireddy, Felix Hans Michel Grimberg

Background. In medicine and other applications, the copying and sharing of data is impractical for a range of well-considered reasons. With federated learning (FL) techniques, machine learning models can be trained on data spread across several locations w ...

Springer, Cham2020

Practical Low-Rank Communication Compression in Decentralized Deep Learning

Martin Jaggi, Sai Praneeth Reddy Karimireddy, Thijs Vogels

Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models. However, algorithms for decentralized training with compressed communication over arbi ...

2020

Accelerating Gradient Boosting Machines

Sai Praneeth Reddy Karimireddy

Gradient Boosting Machine (GBM) introduced by Friedman (2001) is a widely popular ensembling technique and is routinely used in competitions such as Kaggle and the KDD-Cup (Chen and Guestrin, 2016). In this work, we propose an Accelerated Gradient Boosting ...

ADDISON-WESLEY PUBL CO2020

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

Sai Praneeth Reddy Karimireddy, Sebastian Urban Stich

We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show that the rate of convergence in all cases consists of two terms: (i) a stocha ...

Microtome Publishing2020

SCAFFOLD Stochastic Controlled Averaging for Federated Learning

Sai Praneeth Reddy Karimireddy, Sebastian Urban Stich, Ananda Theertha Suresh

Federated Averaging (FEDAVG) has emerged as the algorithm of choice for federated learning due to its simplicity and low communication cost. However, in spite of recent research efforts, its performance is not fully understood. We obtain tight convergence ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2020

Efficient Greedy Coordinate Descent for Composite Problems

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy, Anastasiia Koloskova

Coordinate descent with random coordinate selection is the current state of the art for many large scale optimization problems. However, greedy selection of the steepest coordinate on smooth problems can yield convergence rates independent of the dimension ...

2019

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Martin Jaggi, Sebastian Urban Stich, Quentin Rebjock, Sai Praneeth Reddy Karimireddy

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD doe ...

PMLR2019