Stochastic Processes: Markov Chains

This lecture introduces stochastic processes, focusing on Markov chains. A discrete-time stochastic process is defined, with examples like stock prices and website hits. The concept of Markov chains is explained, where the future evolution depends only on the present state. Transition probabilities and state space are discussed, emphasizing time-homogeneous Markov chains. The lecture covers the formal definition of Markov chains, state distributions, and graphical representations. Various examples, including inventory models and queuing systems, illustrate the application of Markov chains in real-world scenarios.