Variance Reduction in Deep Learning

This lecture delves into variance reduction techniques in deep learning, starting with an exploration of gradient descent versus stochastic gradient descent. The instructor explains how to decrease variance while using a constant step-size and introduces the concept of mini-batch stochastic gradient descent. The lecture covers the Stochastic Variance Reduced Gradient (SVRG) method, convergence analysis, and the performance comparison of different algorithms. It concludes with a discussion on variance reduction for non-convex problems and provides insights into the application of variance reduction techniques in deep learning.