Maximum entropy probability distribution

In statistics and information theory, a maximum entropy probability distribution has entropy that is at least as great as that of all other members of a specified class of probability distributions. According to the principle of maximum entropy, if nothing is known about a distribution except that it belongs to a certain class (usually defined in terms of specified properties or measures), then the distribution with the largest entropy should be chosen as the least-informative default. The motivation is twofold: first, maximizing entropy minimizes the amount of prior information built into the distribution; second, many physical systems tend to move towards maximal entropy configurations over time. If is a discrete random variable with distribution given by then the entropy of is defined as If is a continuous random variable with probability density , then the differential entropy of is defined as The quantity is understood to be zero whenever . This is a special case of more general forms described in the articles Entropy (information theory), Principle of maximum entropy, and differential entropy. In connection with maximum entropy distributions, this is the only one needed, because maximizing will also maximize the more general forms. The base of the logarithm is not important as long as the same one is used consistently: change of base merely results in a rescaling of the entropy. Information theorists may prefer to use base 2 in order to express the entropy in bits; mathematicians and physicists will often prefer the natural logarithm, resulting in a unit of nats for the entropy. The choice of the measure is however crucial in determining the entropy and the resulting maximum entropy distribution, even though the usual recourse to the Lebesgue measure is often defended as "natural". Many statistical distributions of applicable interest are those for which the moments or other measurable quantities are constrained to be constants. The following theorem by Ludwig Boltzmann gives the form of the probability density under these constraints.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (1)

Related concepts (32)

Related courses (32)

Related lectures (340)

Computing the partition function and the marginals of a global probability distribution are two important issues in any probabilistic inference problem. In a previous work, we presented sub-tree based upper and lower bounds on the partition function of a given probabilistic inference problem. Using the entropies of the sub-trees we proved an inequality that compares the lower bounds obtained from different sub-trees. In this paper we investigate the properties of one specific lower bound, namely the lower bound computed by the minimum entropy sub-tree. We also investigate the relationship between the minimum entropy sub-tree and the sub-tree that gives the best lower bound.

Ieee Service Center, 445 Hoes Lane, Po Box 1331, Piscataway, Nj 08855-1331 Usa2008

CS-101: Advanced information, computation, communication I

Discrete mathematics is a discipline with applications to almost all areas of study. It provides a set of indispensable tools to computer science in particular. This course reviews (familiar) topics a

PHYS-231: Data analysis for Physics

Ce cours présentera les bases de l'analyse des données et de l'apprentissage à partir des données, l'estimation des erreurs et la stochasticité en physique. Les concepts seront introduits théoriquemen

EE-559: Deep learning

This course explores how to design reliable discriminative and generative neural networks, the ethics of data acquisition and model deployment, as well as modern multi-modal models.

Laplace distribution

In probability theory and statistics, the Laplace distribution is a continuous probability distribution named after Pierre-Simon Laplace. It is also sometimes called the double exponential distribution, because it can be thought of as two exponential distributions (with an additional location parameter) spliced together along the abscissa, although the term is also sometimes used to refer to the Gumbel distribution.

Differential entropy

Differential entropy (also referred to as continuous entropy) is a concept in information theory that began as an attempt by Claude Shannon to extend the idea of (Shannon) entropy, a measure of average (surprisal) of a random variable, to continuous probability distributions. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the limiting density of discrete points (LDDP).

Maximum entropy probability distribution

Fokker Planck Equation MOOC: Advanced statistical physics

Covers the Fokker-Planck equation and Kramer's Moyal expansion in statistical physics.

Probability Distributions: Understanding Parameters and Transformations MATH-447: Risk, rare events and extremes

Explores probability distributions, parameters, and transformations to analyze events within distributions.

Eigenvalues and EM Algorithm MGT-448: Statistical inference and machine learning

Covers eigenvectors, principal components, likelihood variables, EM algorithm, Jensen's inequality, and maximizing lower bounds.