Are you an EPFL student looking for a semester project?
Work with us on data science and visualisation projects, and deploy your project as an app on top of Graph Search.
Concept
Divergence (statistics)
Summary
In information geometry, a divergence is a kind of statistical distance: a binary function which establishes the separation from one probability distribution to another on a statistical manifold.
The simplest divergence is squared Euclidean distance (SED), and divergences can be viewed as generalizations of SED. The other most important divergence is relative entropy (also called Kullback–Leibler divergence), which is central to information theory. There are numerous other specific divergences and classes of divergences, notably f-divergences and Bregman divergences (see ).
Given a differentiable manifold of dimension , a divergence on is a -function satisfying:
for all (non-negativity),
if and only if (positivity),
At every point , is a positive-definite quadratic form for infinitesimal displacements from .
In applications to statistics, the manifold is typically the space of parameters of a parametric family of probability distributions.
Condition 3 means that defines an inner product on the tangent space for every . Since is on , this defines a Riemannian metric on .
Locally at , we may construct a local coordinate chart with coordinates , then the divergence is where is a matrix of size . It is the Riemannian metric at point expressed in coordinates .
Dimensional analysis of condition 3 shows that divergence has the dimension of squared distance.
The dual divergence is defined as
When we wish to contrast against , we refer to as primal divergence.
Given any divergence , its symmetrized version is obtained by averaging it with its dual divergence:
Unlike metrics, divergences are not required to be symmetric, and the asymmetry is important in applications. Accordingly, one often refers asymmetrically to the divergence "of q from p" or "from p to q", rather than "between p and q". Secondly, divergences generalize squared distance, not linear distance, and thus do not satisfy the triangle inequality, but some divergences (such as the Bregman divergence) do satisfy generalizations of the Pythagorean theorem.
Official source
About this result
This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.