Optimization Techniques: Stochastic Gradient Descent and Beyond

This lecture covers optimization techniques in machine learning, focusing on stochastic gradient descent (SGD) and its applications. It begins with an introduction to constrained optimization, explaining how SGD can be adapted for constrained problems through projected SGD. The instructor discusses the structure of objective functions and the efficiency of SGD compared to full gradient descent, highlighting the cost benefits of using stochastic gradients. The concept of unbiasedness in stochastic gradients is introduced, along with theorems regarding convergence rates under certain conditions. The lecture also explores mini-batch SGD, emphasizing the reduction of variance and the benefits of parallel computation. The discussion extends to non-convex optimization challenges, including local minima and saddle points, and the behavior of gradient descent in these scenarios. Finally, the lecture touches on Hamiltonian mechanics and the transformation of coordinates, linking optimization techniques to broader mathematical concepts.