Concept# Entropy (information theory)

Summary

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable X, which takes values in the alphabet \mathcal{X} and is distributed according to p\colon \mathcal{X}\to[0, 1]:
\Eta(X) := -\sum_{x \in \mathcal{X}} p(x) \log p(x) = \mathbb{E}[-\log p(X)] ,
where \Sigma denotes the sum over the variable's possible values. The choice of base for \log, the logarithm, varies for different applications. Base 2 gives the unit of bits (or "shannons"), while base e gives "natural units" nat, and base 10 gives units of "dits", "bans", or "hartleys". An equivalent definition of entropy is the expected value of the self-information of a variable.
The concept of information entropy was introduced by Claude Shannon in his 1948 paper "A Mathematical Theory of Communic

Official source

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications

Loading

Related people

Loading

Related units

Loading

Related concepts

Loading

Related courses

Loading

Related lectures

Loading

Related publications (100)

Loading

Loading

Loading

Related concepts (97)

Information theory

Information theory is the mathematical study of the quantification, storage, and communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in

Mutual information

In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies th

Entropy

Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in

Related people (15)

Related courses (80)

COM-406: Foundations of Data Science

We discuss a set of topics that are important for the understanding of modern data science but that are typically not taught in an introductory ML course. In particular we discuss fundamental ideas and techniques that come from probability, information theory as well as signal processing.

ME-251: Thermodynamics and energetics I

Introduction aux principes de la thermodynamique, propriétés thermodynamiques de la matière et à leur calcul. Les étudiants maîtriseront les concepts de conservation (chaleur, masse, quantité de mouvement) et appliqueront ces concepts au cycles thermodynamiques et systèmes de conversion d'énergie.

PHYS-105: Advanced physics II (thermodynamics)

Ce cours présente la thermodynamique en tant que théorie permettant une description d'un grand nombre de phénomènes importants en physique, chimie et ingéniere, et d'effets de transport. Une introduction à la physique statistique renforce les notions acquises grâce à une modélisation microscopique.

Related units (14)

Shannon, in his landmark 1948 paper, developed a framework for characterizing the fundamental limits of information transmission. Among other results, he showed that reliable communication over a channel is possible at any rate below its capacity. In 2008, Arikan discovered polar codes; the only class of explicitly constructed low-complexity codes that achieve the capacity of any binary-input memoryless symmetric-output channel. Arikan's polar transform turns independent copies of a noisy channel into a collection of synthetic almost-noiseless and almost-useless channels. Polar codes are realized by sending data bits over the almost-noiseless channels and recovering them by using a low-complexity successive-cancellation (SC) decoder, at the receiver. In the first part of this thesis, we study polar codes for communications. When the underlying channel is an erasure channel, we show that almost all correlation coefficients between the erasure events of the synthetic channels decay rapidly. Hence, the sum of the erasure probabilities of the information-carrying channels is a tight estimate of the block-error probability of polar codes when used for communication over the erasure channel. We study SC list (SCL) decoding, a method for boosting the performance of short polar codes. We prove that the method has a numerically stable formulation in log-likelihood ratios. In hardware, this formulation increases the decoding throughput by 53% and reduces the decoder's size about 33%. We present empirical results on the trade-off between the length of the CRC and the performance gains in a CRC-aided version of the list decoder. We also make numerical comparisons of the performance of long polar codes under SC decoding with that of short polar codes under SCL decoding. Shannon's framework also quantifies the secrecy of communications. Wyner, in 1975, proposed a model for communications in the presence of an eavesdropper. It was shown that, at rates below the secrecy capacity, there exist reliable communication schemes in which the amount of information leaked to the eavesdropper decays exponentially in the block-length of the code. In the second part of this thesis, we study the rate of this decay. We derive the exact exponential decay rate of the ensemble-average of the information leaked to the eavesdropper in Wyner's model when a randomly constructed code is used for secure communications. For codes sampled from the ensemble of i.i.d. random codes, we show that the previously known lower bound to the exponent is exact. Our ensemble-optimal exponent for random constant-composition codes improves the lower bound extant in the literature. Finally, we show that random linear codes have the same secrecy power as i.i.d. random codes. The key to securing messages against an eavesdropper is to exploit the randomness of her communication channel so that the statistics of her observation resembles that of a pure noise process for any sent message. We study the effect of feedback on this approximation and show that it does not reduce the minimum entropy rate required to approximate a given process. However, we give examples where variable-length schemes achieve much larger exponents in this approximation in the presence of feedback than the exponents in systems without feedback. Upper-bounding the best exponent that block codes attain, we conclude that variable-length coding is necessary for achieving the improved exponents.

Linlin Cao, Ning Chu, Shuai Yang

In this study, a double volute centrifugal pump with relative low efficiency and high vibration is redesigned to improve the efficiency and reduce the unsteady radial forces with the aid of unsteady computational fluid dynamics (CFD) analysis. The concept of entropy generation rate is applied to evaluate the magnitude and distribution of the loss generation in pumps and it is proved to be a useful technique for loss identification and subsequent redesign process. The local Euler head distribution (LEHD) can represent the energy growth from the blade leading edge (LE) to its trailing edge (TE) on constant span stream surface in a viscous flow field, and the LEHD is proposed to evaluate the flow field on constant span stream surfaces from hub to shroud. To investigate the unsteady internal flow of the centrifugal pump, the unsteady Reynolds-Averaged Navier-Stokes equations (URANS) are solved with realizable k-epsilon turbulence model using the CFD code FLUENT. The impeller is redesigned with the same outlet diameter as the baseline pump. A two-step-form LEHD is recommended to suppress flow separation and secondary flow encountered in the baseline impeller in order to improve the efficiency. The splitter blades are added to improve the hydraulic performance and to reduce unsteady radial forces. The original double volute is substituted by a newly designed single volute one. The hydraulic efficiency of the centrifugal pump based on redesigned impeller with splitter blades and newly designed single volute is about 89.2%, a 3.2% higher than the baseline pump. The pressure fluctuation in the volute is significantly reduced, and the mean and maximum values of unsteady radial force are only 30% and 26.5% of the values for the baseline pump.

Pushed by the proliferation of antennas and of multiuser scenarios, matrices with random entries are appearing more and more frequently in information theory. This leads to the study of matrix channels, where the capacity depends on the distribution of the matrix's eigenvalues. These eigenvalues are complicated functionals of the entries of the matrix and the challenge lies therein. It is often the case that in order to better model different communication scenarios, one is driven away from matrix models typically studied in pure mathematics and physics. One cannot simply resort to the standard tools developed over the years in these fields and must come up with new approaches. In this thesis, our goal is to obtain results in scenarios where the randomness is limited by the nature of the channel, in order to widen applicability in real life scenarios.

Related lectures (164)