Publication
Mutual Information Disentangles Interactions from Changing Environments
Publications associées (32)
Mutual information in changing environments: Nonlinear interactions, out-of-equilibrium systems, and continuously varying diffusivities
Daniel Maria Busiello, Giorgio Nicoletti
Biochemistry, ecology, and neuroscience are examples of prominent fields aiming at describing interacting systems that exhibit nontrivial couplings to complex, ever-changing environments. We have recently shown that linear interactions and a switching envi ...
AMER PHYSICAL SOC2022A Functional Perspective on Information Measures
Since the birth of Information Theory, researchers have defined and exploited various information measures, as well as endowed them with operational meanings. Some were born as a "solution to a problem", like Shannon's Entropy and Mutual Information. Other ...
EPFL2022The Gray-Wyner Network and Wyner's Common Information for Gaussian Sources
Michael Christoph Gastpar, Erixhen Sula
This paper presents explicit solutions for two related non-convex information extremization problems due to Gray and Wyner in the Gaussian case. The first problem is the Gray-Wyner network subject to a sum-rate constraint on the two private links. Here, ou ...
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022A Comparison of optimal measurement-system design with engineering judgement for bridge load testing
Due to conservative approaches in construction design and practice, infrastructure often has hidden reserve capacity. When quantified, this reserve has potential to improve decisions related to asset management. Field measurements, collected through load t ...
2021Robust Generalization via $\alpha$-Mutual Information
Michael Christoph Gastpar, Amedeo Roberto Esposito, Ibrahim Issa
The aim of this work is to provide bounds connecting two probability measures of the same event using Rényi -Divergences and Sibson’s -Mutual Information, a generalization of respectively the Kullback-Leibler Divergence and Shannon’s Mutual ...
ETHZ2020Strengthened Information-theoretic Bounds on the Generalization Error
Michael Christoph Gastpar, Amedeo Roberto Esposito, Ibrahim Issa
The following problem is considered: given a joint distribution P XY and an event E, bound P XY (E) in terms of P X P Y (E) (where P X P Y is the product of the marginals of P XY ) and a measure of dependence of X and Y. Such bounds have direct application ...
IEEE2019Mutual Information for the Stochastic Block Model by the Adaptive Interpolation Method
Nicolas Macris, Jean François Emmanuel Barbier, Chun Lam Chan
We rigorously derive a single-letter variational expression for the mutual information of the asymmetric two-groups stochastic block model in the dense graph regime. Existing proofs in the literature are indirect, as they involve mapping the model to a ran ...
IEEE2019Entropy and mutual information in models of deep neural networks
Nicolas Macris, Florent Gérard Krzakala, Lenka Zdeborová, Jean François Emmanuel Barbier, Clément Dominique Luneau
We examine a class of stochastic deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) we show how entropies and mutual informations can be derived from heuristic statistical physics ...
IOP PUBLISHING LTD2019Entropy and mutual information in models of deep neural networks
Nicolas Macris, Florent Gérard Krzakala, Lenka Zdeborová, Jean François Emmanuel Barbier, Clément Dominique Luneau
We examine a class of stochastic deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics ...
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)2018High-Dimensional Inference on Dense Graphs with Applications to Coding Theory and Machine Learning
We are living in the era of "Big Data", an era characterized by a voluminous amount of available data. Such amount is mainly due to the continuing advances in the computational capabilities for capturing, storing, transmitting and processing data. However, ...
EPFL2018