Publication

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Related publications (32)

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Towards neural network approaches for point cloud compression

Touradj Ebrahimi, Evangelos Alexiou, Kuan Tung

Point cloud imaging has emerged as an efficient and popular solution to represent immersive visual information. However, the large volume of data generated in the acquisition process reveals the need of efficient compression solutions in order to store and ...

2020

Supplementary Material - AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

Sabine Süsstrunk, Majed El Helou, Frederike Dümbgen

In this supplementary material, we present the details of the neural network architecture and training settings used in all our experiments. This holds for all experiments presented in the main paper as well as in this supplementary material. We also show ...

2020

AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

Sabine Süsstrunk, Majed El Helou, Frederike Dümbgen

The large capacity of neural networks enables them to learn complex functions. To avoid overfitting, networks however require a lot of training data that can be expensive and time-consuming to collect. A common practical approach to attenuate overfitting i ...

IEEE2020

We experimentally solve the problem of maximizing capacity under a total supply power constraint in a massively parallel submarine cable context, i.e., for a spatially uncoupled system in which fiber Kerr nonlinearity is not a dominant limitation. By using ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020

p>We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the overparametrized regime where the layer width m is larger than the input dimension d. We conside ...

Curran Associates, Inc.2020

Volumetric Transformer Networks

Sabine Süsstrunk, Mathieu Salzmann, Seungryong Kim

Existing techniques to encode spatial invariance within deep convolutional neural networks (CNNs) apply the same warping field to all the feature channels. This does not account for the fact that the individual feature channels can represent different sema ...

2020

, , ,

We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz constant of the input-output relation. To that end, ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Martin Jaggi, Sebastian Urban Stich, Quentin Rebjock, Sai Praneeth Reddy Karimireddy

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD doe ...

2019

Evaluating and Interpreting Deep Convolutional Neural Networks via Non-negative Matrix Factorization

Edo Collins

With ever greater computational resources and more accessible software, deep neural networks have become ubiquitous across industry and academia. Their remarkable ability to generalize to new samples defies the conventional view, which holds that complex, ...

EPFL2019

Objective quality assessment of compressed images is very useful in many applications. In this paper we present an objective quality metric that is better tuned to evaluate the quality of images distorted by compression artifacts. A deep convolutional neur ...

2019