Deep Reinforcement Learning: Mini-Batches and Policy Methods

This lecture covers the classification of deep reinforcement learning methods, focusing on mini-batches in both on-policy and off-policy contexts. It begins with an overview of deep RL algorithms, including model-free and model-based approaches, and highlights the importance of using independent and identically distributed mini-batches for training. The instructor explains the issues caused by temporally correlated weight updates, which can lead to instabilities in learning. Proposed solutions include using replay buffers and multiple parallel actors to sample data effectively. The lecture also delves into specific algorithms such as Deep Q-Networks (DQN) and Advantage Actor-Critic (A2C), discussing their advantages and disadvantages in terms of sample complexity. The discussion extends to continuous control methods like Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient (DDPG), as well as model-based approaches like AlphaZero and MuZero. The lecture concludes with a quiz to reinforce the concepts covered, ensuring a comprehensive understanding of deep reinforcement learning techniques.