Variance Reduction Techniques

This lecture covers variance reduction techniques in optimization, focusing on gradient descent (GD) and stochastic gradient descent (SGD) methods. The instructor explains how to decrease variance while using a constant step-size, introducing concepts like Lipschitz smoothness and the observation of GD vs. SGD steps. The lecture also delves into the mathematics of data, from theory to computation, with a special emphasis on deep learning. Various algorithms and methods, such as SVRG and mini-batch SGD, are discussed to reduce the variance of stochastic gradients. The lecture explores the challenges in deep learning theory and applications, including fairness, robustness, interpretability, and energy efficiency.