Kullback–Leibler divergence | EPFL Graph Search

Are you an EPFL student looking for a semester project?

Work with us on data science and visualisation projects, and deploy your project as an app on top of GraphSearch.

In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted , is a type of statistical distance: a measure of how one probability distribution P is different from a second, reference probability distribution Q. A simple interpretation of the KL divergence of P from Q is the expected excess surprise from using Q as a model when the actual distribution is P. While it is a distance, it is not a metric, the most familiar type of distance: it is not symmetric in the two distributions (in contrast to variation of information), and does not satisfy the triangle inequality. Instead, in terms of information geometry, it is a type of divergence, a generalization of squared distance, and for certain classes of distributions (notably an exponential family), it satisfies a generalized Pythagorean theorem (which applies to squared distances). In the simple case, a relative entropy of 0 indicates that the two distributions in question have identical quantities of information. Relative entropy is a nonnegative function of two distributions or measures. It has diverse applications, both theoretical, such as characterizing the relative (Shannon) entropy in information systems, randomness in continuous time-series, and information gain when comparing statistical models of inference; and practical, such as applied statistics, fluid mechanics, neuroscience and bioinformatics. Consider two probability distributions and . Usually, represents the data, the observations, or a measured probability distribution. Distribution represents instead a theory, a model, a description or an approximation of . The Kullback–Leibler divergence is then interpreted as the average difference of the number of bits required for encoding samples of using a code optimized for rather than one optimized for . Note that the roles of and can be reversed in some situations where that is easier to compute, such as with the expectation–maximization algorithm and evidence lower bound computations.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (1)

Related concepts (29)

Related courses (41)

Related lectures (454)

Many-body localization in a fragmented Hilbert space

Loïc Jean Pierre Herviou

We study many-body localization (MBL) in a pair-hopping model exhibiting strong fragmentation of the Hilbert space. We show that several Krylov subspaces have both ergodic statistics in the thermodynamic limit and a dimension that scales much slower than the full Hilbert space but still exponentially. Such a property allows us to study the MBL phase transition in systems including up to 64 spins. The different Krylov spaces that we consider show clear signatures of a many-body localization transition, both in the Kullback-Leibler divergence of the distribution of their level spacing ratio and their entanglement properties. However, they also present distinct scalings with the system size. Depending on the subspace, the critical disorder strength can be nearly independent of the system size or conversely show an approximately linear increase with the number of spins.

2021

Kullback–Leibler divergence

Statistical distance

In statistics, probability theory, and information theory, a statistical distance quantifies the distance between two statistical objects, which can be two random variables, or two probability distributions or samples, or the distance can be between an individual sample point and a population or a wider sample of points. A distance between populations can be interpreted as measuring the distance between two probability distributions and hence they are essentially measures of distances between probability measures.

Divergence (statistics)

In information geometry, a divergence is a kind of statistical distance: a binary function which establishes the separation from one probability distribution to another on a statistical manifold. The simplest divergence is squared Euclidean distance (SED), and divergences can be viewed as generalizations of SED. The other most important divergence is relative entropy (also called Kullback–Leibler divergence), which is central to information theory.

COM-406: Foundations of Data Science

We discuss a set of topics that are important for the understanding of modern data science but that are typically not taught in an introductory ML course. In particular we discuss fundamental ideas an

BIO-369: Randomness and information in biological data

Biology is becoming more and more a data science, as illustrated by the explosion of available genome sequences. This course aims to show how we can make sense of such data and harness it in order to

PHYS-467: Machine learning for physicists

Machine learning and data analysis are becoming increasingly central in sciences including physics. In this course, fundamental principles and methods of machine learning will be introduced and practi

Quantum Information PHYS-758: Advanced Course on Quantum Communication

Explores the CHSH operator, self-testing, eigenstates, and quantifying randomness in quantum systems.

Quantum Source Coding PHYS-758: Advanced Course on Quantum Communication

Covers entropic notions in quantum sources, Shannon entropy, Von Neumann entropy, and source coding.

Superficial Degree of Divergence

Explores the concept of superficial degree of divergence and its mathematical implications.