Lecture

Why are there so many saddle points?: Loss landscape and optimization methods

Description

This lecture by the instructor explores the reasons behind the abundance of saddle points in deep learning optimization. Starting with the statistical argument based on the Hessian matrix, it delves into the geometric argument involving permutations. The lecture discusses the relationship between minima and saddle points, emphasizing the statistical and modern views. It explains how the weight space symmetry contributes to the prevalence of saddle points and presents examples to illustrate these concepts. The geometric argument and weight space symmetry are further elaborated to demonstrate the abundance of saddle points compared to global minima. The lecture concludes by summarizing the loss landscape in deep neural networks, highlighting the presence of multiple minima and saddle points.

About this result
This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.