Stochastic Gradient Descent

This lecture covers the concept of stochastic gradient descent, where the algorithm chooses an initial point and updates it using stochastic gradients. It discusses unbiasedness, convexity, and the comparison with full gradient descent. The lecture also explores the convergence rate, bounded stochastic gradients, and the implications of strong convexity. Additionally, it delves into the challenges of gradient descent in the non-convex world and the benefits of mini-batch stochastic gradient descent. The content extends to smooth functions, bounded Hessians, and the convergence of gradient descent on smooth functions. The lecture concludes with insights on the behavior of gradient descent in non-convex optimization.