Conditional entropy

Applied sciences
Information engineering
Information theory
Channel capacity

Related publications (23)

About
Privacy
Disclaimer

Graph Chatbot

Differential Entropy of the Conditional Expectation Under Additive Gaussian Noise

Michael Christoph Gastpar, Alper Köse, Ahmet Arda Atalik

The conditional mean is a fundamental and important quantity whose applications include the theories of estimation and rate-distortion. It is also notoriously difficult to work with. This paper establishes novel bounds on the differential entropy of the co ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

The Gray-Wyner Network and Wyner's Common Information for Gaussian Sources

Michael Christoph Gastpar, Erixhen Sula

This paper presents explicit solutions for two related non-convex information extremization problems due to Gray and Wyner in the Gaussian case. The first problem is the Gray-Wyner network subject to a sum-rate constraint on the two private links. Here, ou ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Differential Entropy of the Conditional Expectation under Gaussian Noise

Michael Christoph Gastpar, Alper Köse, Ahmet Arda Atalik

This paper considers an additive Gaussian noise channel with arbitrarily distributed finite variance input signals. It studies the differential entropy of the minimum mean-square error (MMSE) estimator and provides a new lower bound which connects the diff ...

IEEE2021

Mutual Information and Optimality of Approximate Message-Passing in Random Linear Estimation

Nicolas Macris, Jean François Emmanuel Barbier, Florent Gérard Krzakala, Mohamad Baker Dia

We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections. A few examples where this problem is relevant are compressed sensing, sparse superposition codes, and code division multiple access. There has been a ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020

Reed-Muller codes polarize

Emmanuel Abbé, Min Ye

Reed-Muller (RM) codes were introduced in 1954 and have long been conjectured to achieve Shannon's capacity on symmetric channels. The activity on this conjecture has recently been revived with the emergence of polar codes. RM codes and polar codes are gen ...

IEEE COMPUTER SOC2019

Dimension-free Information Concentration via Exp-Concavity

Volkan Cevher, Ya-Ping Hsieh

Information concentration of probability measures have important implications in learning theory. Recently, it is discovered that the information content of a log-concave distribution concentrates around their differential entropy, albeit with an unpleasan ...

2018

Chaining Mutual Information and Tightening Generalization Bounds

Emmanuel Abbé

Bounding the generalization error of learning algorithms has a long history, which yet falls short in explaining various generalization successes including those of deep learning. Two important difficulties are (i) exploiting the dependencies between the h ...

NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)2018

Lower Bounds on Active Learning for Graphical Model Selection

Volkan Cevher, Jonathan Mark Scarlett

We consider the problem of estimating the underlying graph associated with a Markov random field, with the added twist that the decoding algorithm can iteratively choose which subsets of nodes to sample based on the previous samples, resulting in an active ...

2017

Spatial Coupling as a Proof Technique and Three Applications

Rüdiger Urbanke, Nicolas Macris, Andrei Giurgiu

The aim of this paper is to show that spatial coupling can be viewed not only as a means to build better graphical models, but also as a tool to better understand uncoupled models. The starting point is the observation that some asymptotic properties of gr ...

Ieee-Inst Electrical Electronics Engineers Inc2016

The Bethe Free Energy Allows to Compute the Conditional Entropy of Graphical Code Instances: A Proof From the Polymer Expansion

Nicolas Macris, Marc Vuffray

The main objective of this paper is to explore the precise relationship between the Bethe free energy (or entropy) and the Shannon conditional entropy of graphical error correcting codes. The main result shows that the Bethe free energy associated with a l ...

Ieee-Inst Electrical Electronics Engineers Inc2016

Page 1 of 3

The Bethe Free Energy Allows to Compute the Conditional Entropy of Graphical Code Instances: A Proof From the Polymer Expansion

Nicolas Macris, Marc Vuffray

Ieee-Inst Electrical Electronics Engineers Inc2016

Mutual Information and Optimality of Approximate Message-Passing in Random Linear Estimation

Nicolas Macris, Jean François Emmanuel Barbier, Florent Gérard Krzakala, Mohamad Baker Dia

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020