Why are there so many saddle points?: Loss landscape and optimization methods

This lecture by the instructor explores the reasons behind the abundance of saddle points in deep learning optimization. Starting with the statistical argument based on the Hessian matrix, it delves into the geometric argument involving permutations. The lecture discusses the relationship between minima and saddle points, emphasizing the statistical and modern views. It explains how the weight space symmetry contributes to the prevalence of saddle points and presents examples to illustrate these concepts. The geometric argument and weight space symmetry are further elaborated to demonstrate the abundance of saddle points compared to global minima. The lecture concludes by summarizing the loss landscape in deep neural networks, highlighting the presence of multiple minima and saddle points.