Structures in Non-Convex Optimization

This lecture delves into scalable non-convex optimization, focusing on deep learning. It covers the optimization formulation for deep-learning training problems, the challenges faced in training neural networks, and the critical points classification. The instructor discusses the strict saddle property, the convergence of Stochastic Gradient Descent (SGD), and the avoidance of saddle points. Additionally, the lecture explores the convergence speed to local minimizers, the impact of step-size decrease in practice, and the phenomena related to neural networks and overparametrization. Various adaptive first-order methods like AdaGrad and RMSProp are explained, emphasizing their role in optimizing gradient descent.