Lecture

Gradient Descent: Optimization Techniques

Description

This lecture covers the concepts of gradient descent, convex and non-convex loss functions, stochastic gradient descent, and early stopping in the context of neural networks training. It explains the importance of small weights at the beginning of gradient descent, the impact of validation loss increase, and the norm of parameters during training. The lecture also delves into the differences between standard and stochastic gradient descent, emphasizing the computational efficiency of the latter. Various optimization techniques and strategies are discussed, including the use of ADAMW optimizer and the concept of early stopping as a form of regularization.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace
About this result
This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.