Publications related to Interaction information

Entropy and mutual information in models of deep neural networks

Nicolas Macris, Florent Gérard Krzakala, Lenka Zdeborová, Jean François Emmanuel Barbier, Clément Dominique Luneau

We examine a class of stochastic deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) we show how entropies and mutual informations can be derived from heuristic statistical physics ...

IOP PUBLISHING LTD2019

Theoretical foundations of forward feature selection methods based on mutual information

Francisco Santos Paredes Quartin de Macedo

Feature selection problems arise in a variety of applications, such as microarray analysis, clinical prediction, text categorization, image classification and face recognition, multi-label learning, and classification of internet traffic. Among the various ...

ELSEVIER SCIENCE BV2019

Mutual Information for Low-Rank Even-Order Symmetric Tensor Factorization

Nicolas Macris, Jean François Emmanuel Barbier, Clément Dominique Luneau

We consider a statistical model for finite-rank symmetric tensor factorization and prove a single-letter variational expression for its mutual information when the tensor is of even order. The proof uses the adaptive interpolation method, for which rank-on ...

IEEE2019

Quantifying high-order interdependencies via multivariate extensions of the mutual information

Michael Christoph Gastpar, Fernando Rosas de Andraca

This paper introduces a model-agnostic approach to study statistical synergy, a form of emergence in which patterns at large scales are not traceable from lower scales. Our framework leverages various multivariate extensions of Shannon's mutual information ...

2019

Subadditivity Beyond Trees and the Chi-Squared Mutual Information

Emmanuel Abbé, Enric Boix Adsera

Evans et al. [1] proved the subadditivity of the mutual information in the broadcasting on tree model with binary vertex labels and symmetric edge channels. They raised the question of whether such subadditivity extends to loopy graphs in some appropriate ...

IEEE2019

Learning and Adaptive Data Analysis via Maximal Leakage

Michael Christoph Gastpar, Amedeo Roberto Esposito, Ibrahim Issa

There has been growing interest in studying connections between generalization error of learning algorithms and information measures. In this work, we generalize a result that employs the maximal leakage, a measure of leakage of information, and explore ho ...

IEEE2019

Optimal errors and phase transitions in high-dimensional generalized linear models

Nicolas Macris, Florent Gérard Krzakala, Lenka Zdeborová, Jean François Emmanuel Barbier

Generalized linear models (GLMs) are used in high-dimensional machine learning, statistics, communications, and signal processing. In this paper we analyze GLMs when the data matrix is random, as relevant in problems such as compressed sensing, error-corre ...

PNAS2019

Reed-Muller codes polarize

Emmanuel Abbé, Min Ye

Reed-Muller (RM) codes were introduced in 1954 and have long been conjectured to achieve Shannon's capacity on symmetric channels. The activity on this conjecture has recently been revived with the emergence of polar codes. RM codes and polar codes are gen ...

IEEE COMPUTER SOC2019

The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference

Nicolas Macris, Jean François Emmanuel Barbier

In recent years important progress has been achieved towards proving the validity of the replica predictions for the (asymptotic) mutual information (or free energy) in Bayesian inference problems. The proof techniques that have emerged appear to be quite ...

SPRINGER HEIDELBERG2019

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Interaction information

Entropy and mutual information in models of deep neural networks

Theoretical foundations of forward feature selection methods based on mutual information

Mutual Information for Low-Rank Even-Order Symmetric Tensor Factorization

Measuring transient phase-amplitude coupling using local mutual information

Quantifying high-order interdependencies via multivariate extensions of the mutual information

Subadditivity Beyond Trees and the Chi-Squared Mutual Information

Learning and Adaptive Data Analysis via Maximal Leakage

Optimal errors and phase transitions in high-dimensional generalized linear models

Reed-Muller codes polarize

The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference

Graph Chatbot

Chat with Graph Search

Measuring transient phase-amplitude coupling using local mutual information

Subadditivity Beyond Trees and the Chi-Squared Mutual Information

Reed-Muller codes polarize

Quantifying high-order interdependencies via multivariate extensions of the mutual information

The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference

Optimal errors and phase transitions in high-dimensional generalized linear models

Theoretical foundations of forward feature selection methods based on mutual information

Entropy and mutual information in models of deep neural networks

Learning and Adaptive Data Analysis via Maximal Leakage

Mutual Information for Low-Rank Even-Order Symmetric Tensor Factorization