Eligibility Traces for Policy Gradient and Actor-Critic

This lecture covers the combination of policy-gradient learning and actor-critic architectures with eligibility traces, leading to an elegant online learning rule. It explains how the actor learns through policy gradient while the critic learns via TD-learning, updating eligibility traces and weights accordingly. The lecture also reviews the concept of eligibility traces, their decay over time, and their role in updating Q-values. Furthermore, it explores the use of eligibility traces in policy gradient, keeping memory of previous candidate updates and updating parameters of the 'actor' network. The schematics of Actor-Critic with Eligibility Traces are presented, illustrating the parameters and actions involved. The lecture concludes with an algorithm for estimating values using differentiable policy and state-value function parameterizations.