Publications de Lingjing Kong | EPFL Graph Search

Consensus Control for Decentralized Deep Learning

Martin Jaggi, Tao Lin, Sebastian Urban Stich, Anastasiia Koloskova, Lingjing Kong

Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers fro ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems

Boi Faltings, Fei Mi, Lingjing Kong, Fengyu Cai, Wanhao Zhou

As the labeling cost for different modules in task-oriented dialog (ToD) systems is expensive, a major challenge is to train different modules with the least amount of labeled data. Recently, large-scale pre-trained language models, have shown promising re ...

ASSOC COMPUTATIONAL LINGUISTICS-ACL2021

Extrapolation for Large-batch Training in Deep Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batc ...

2020

Ensemble Distillation for Robust Model Fusion in Federated Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the par ...

2020