Deep Learning III

This lecture covers the optimization formulation of deep learning training problems, the challenges faced in training neural networks, and the concepts of Stochastic Gradient Descent (SGD) and its variants. It also discusses critical points, the strict saddle property, and the convergence of SGD to critical points. Additionally, it explores the optimization landscape of overparametrized neural networks, the phenomenon of overparametrization, and stochastic adaptive first-order methods. The lecture concludes with a detailed explanation of the Variable Metric Stochastic Gradient Descent Algorithm and Adaptive Gradient Methods.