Model-Free Prediction in Reinforcement Learning: Key Methods

This lecture introduces model-free prediction methods in reinforcement learning, focusing on estimating value functions without knowledge of transition dynamics. The instructor begins by contrasting model-based and model-free approaches, emphasizing the importance of learning from experience. The discussion covers two primary methods: Monte Carlo and Temporal Differences (TD). Monte Carlo methods estimate value functions by averaging returns from sampled trajectories, while TD methods update value estimates incrementally based on immediate rewards and subsequent estimates. The lecture also explores the stochastic gradient descent (SGD) algorithm as a means to optimize these estimates, highlighting the role of unbiased gradient estimators. The instructor explains the challenges of infinite horizon problems and how TD methods can address these issues. The session concludes with a summary of the differences between Monte Carlo and TD methods, including their biases and variances, and the implications for policy evaluation and improvement in reinforcement learning.