Publication

On Maintaining Linear Convergence of Distributed Learning and Optimization Under Limited Communication

Abstract

In distributed optimization and machine learning, multiple nodes coordinate to solve large problems. To do this, the nodes need to compress important algorithm information to bits so that it can be communicated over a digital channel. The communication time of these algorithms follows a complex interplay between a) the algorithm's convergence properties, b) the compression scheme, and c) the transmission rate offered by the digital channel. We explore these relationships for a general class of linearly convergent distributed algorithms. In particular, we illustrate how to design quantizers for these algorithms that compress the communicated information to a few bits while still preserving the linear convergence. Moreover, we characterize the communication time of these algorithms as a function of the available transmission rate. We illustrate our results on learning algorithms using different communication structures, such as decentralized algorithms where a single master coordinates information from many workers and fully distributed algorithms where only neighbours in a communication graph can communicate. We conclude that a co-design of machine learning and communication protocols are mandatory to flourish machine learning over networks.

About this result
This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.
Related concepts (34)
Communication
Communication is usually defined as the transmission of information. The term can also refer to the message itself, or the field of inquiry studying these transmissions, also known as communication studies. The precise definition of communication is disputed. Controversial issues are whether unintentional or failed transmissions are included and whether communication does not just transmit meaning but also create it. Models of communication aim to provide a simplified overview of its main components and their interaction.
Organizational communication
Within the realm of communication studies, organizational communication is a field of study surrounding all areas of communication and information flow that contribute to the functioning of an organization . Organizational communication is constantly evolving and as a result, the scope of organizations included in this field of research have also shifted over time. Now both traditionally profitable companies, as well as NGO's and non-profit organizations, are points of interest for scholars focused on the field of organizational communication.
Interpersonal communication
Interpersonal communication is an exchange of information between two or more people. It is also an area of research that seeks to understand how humans use verbal and nonverbal cues to accomplish a number of personal and relational goals. Interpersonal communication research addresses at least six categories of inquiry: 1) how humans adjust and adapt their verbal communication and nonverbal communication during face-to-face communication; 2) how messages are produced; 3) how uncertainty influences behavior and information-management strategies; 4) deceptive communication; 5) relational dialectics; and 6) social interactions that are mediated by technology.
Show more
Related publications (91)

Hypergraph-Based Fast Distributed AC Power Flow Optimization

Colin Neil Jones, Yuning Jiang, Yingzhao Lian, Xinliang Dai

This paper presents a novel distributed approach for solving AC power flow (PF) problems. The optimization problem is reformulated into a distributed form using a communication structure corresponding to a hypergraph, by which complex relationships between ...
New York2023

Communication-efficient distributed training of machine learning models

Thijs Vogels

In this thesis, we explore techniques for addressing the communication bottleneck in data-parallel distributed training of deep learning models. We investigate algorithms that either reduce the size of the messages that are exchanged between workers, or th ...
EPFL2023

The evolution of behavioral cues and signaling in displaced communication

Dario Floreano, Laurent Keller

Displaced communication, whereby individuals communicate regarding a subject that is not immediately present (spatially or temporally), is one of the key features of human language. It also occurs in a few animal species, most notably the honeybee, where t ...
2023
Show more
Related MOOCs (14)
Information, Calcul, Communication: Introduction à la pensée informatique
Dans une première partie, nous étudierons d’abord comment résoudre de manière très concrète un problème au moyen d’un algorithme, ce qui nous amènera dans un second temps à une des grandes questions d
Information, Calcul, Communication: Introduction à la pensée informatique
Dans une première partie, nous étudierons d’abord comment résoudre de manière très concrète un problème au moyen d’un algorithme, ce qui nous amènera dans un second temps à une des grandes questions d
Digital Signal Processing I
Basic signal processing concepts, Fourier analysis and filters. This module can be used as a starting point or a basic refresher in elementary DSP
Show more