Stationary Distribution in Markov Chains

This lecture covers examples of stationary distribution in Markov chains, focusing on the cyclic random walk. It explains the concept of positive-recurrent chains, the existence and uniqueness of the stationary distribution, and provides proofs and counter-examples. The lecture also discusses the case of irreducible and reducible chains, highlighting the implications on the existence and uniqueness of the stationary distribution. Additionally, it explores the scenario where the chain is not irreducible, leading to the presence of multiple transient classes. The importance of understanding the stationary distribution in Markov chains is emphasized throughout the lecture.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (33)

Related concepts (34)

COM-516: Markov chains and algorithmic applications

The study of random walks finds many applications in computer science and communications. The goal of the course is to get familiar with the theory of random walks, and to get an overview of some appl

Markov chain

A Markov chain or Markov process is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. Informally, this may be thought of as, "What happens next depends only on the state of affairs now." A countably infinite sequence, in which the chain moves state at discrete time steps, gives a discrete-time Markov chain (DTMC). A continuous-time process is called a continuous-time Markov chain (CTMC).

Markov model

In probability theory, a Markov model is a stochastic model used to model pseudo-randomly changing systems. It is assumed that future states depend only on the current state, not on the events that occurred before it (that is, it assumes the Markov property). Generally, this assumption enables reasoning and computation with the model that would otherwise be intractable. For this reason, in the fields of predictive modelling and probabilistic forecasting, it is desirable for a given model to exhibit the Markov property.

Markov property

In probability theory and statistics, the term Markov property refers to the memoryless property of a stochastic process, which means that its future evolution is independent of its history. It is named after the Russian mathematician Andrey Markov. The term strong Markov property is similar to the Markov property, except that the meaning of "present" is defined in terms of a random variable known as a stopping time. The term Markov assumption is used to describe a model where the Markov property is assumed to hold, such as a hidden Markov model.

Hidden Markov model

A hidden Markov model (HMM) is a statistical Markov model in which the system being modeled is assumed to be a Markov process — call it — with unobservable ("hidden") states. As part of the definition, HMM requires that there be an observable process whose outcomes are "influenced" by the outcomes of in a known way.

Stationary distribution

Stationary distribution may refer to: A special distribution for a Markov chain such that if the chain starts with its stationary distribution, the marginal distribution of all states at any time will always be the stationary distribution. Assuming irreducibility, the stationary distribution is always unique if it exists, and its existence can be implied by positive recurrence of all states. The stationary distribution has the interpretation of the limiting distribution when the chain is irreducible and aperiodic.

Markov Chains: Transition Densities FIN-415: Probability and stochastic calculus

Covers Markov processes, transition densities, and distribution conditional on information, discussing classification of states and stationary distributions.

Markov Chains: Definitions and Transitions

Explains Markov chains, transition matrices, and stationary distributions in random processes.

Optimal Selling Strategy MGT-484: Applied probability & stochastic processes

Covers the optimal selling strategy in a dynamic scenario to maximize profit.

NISQ and IBM Q COM-309: Introduction to quantum information processing

Explores NISQ devices and IBM Q, covering noisy quantum circuits, qubit technologies, and quantum algorithm development.

Coupling of Markov Chains: Ergodic Theorem COM-516: Markov chains and algorithmic applications

Explores the coupling of Markov chains and the proof of the ergodic theorem, emphasizing distribution convergence and chain properties.