Reinforcement Learning: Exploration, Credit Assignment, TRPO, PPO

This lecture covers the main problems in Reinforcement Learning (RL), focusing on Exploration/Exploitation and Credit Assignment. It discusses the challenges of optimizing policies, emphasizing the need for balancing exploration and exploitation. The instructor explains the concepts of TRPO and PPO for monotonic improvement in RL, highlighting the importance of preventing overfitting to sampled trajectories. Additionally, the limitations of RL are explored, including the safety concerns of exploration in non-simulated settings and the difficulty of credit assignment in sparse-reward scenarios. The lecture concludes with a discussion on the trade-offs of using simulators in RL training and the implications for real-world applications.