Optimization: Gradient Descent and Subgradients

This lecture delves into optimization methods for training machine learning models, focusing on gradient descent and subgradients. The instructor explains the iterative process of minimizing loss functions using naive search, gradient descent, and stochastic gradient descent. The lecture covers the concept of subgradients for non-differentiable functions, providing insights into the linear models' optimization process. Additionally, the instructor introduces advanced optimization techniques like Adam optimization and discusses the importance of parallelization in optimizing large-scale models.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (1,000)

Related concepts (247)

CS-433: Machine learning

Machine learning methods are becoming increasingly central in many sciences and applications. In this course, fundamental principles and methods of machine learning will be introduced, analyzed and pr

Stochastic gradient descent

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data).

Regression analysis

In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance) and one or more independent variables (often called 'predictors', 'covariates', 'explanatory variables' or 'features'). The most common form of regression analysis is linear regression, in which one finds the line (or a more complex linear combination) that most closely fits the data according to a specific mathematical criterion.

Gradient descent

In mathematics, gradient descent (also often called steepest descent) is a iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a local maximum of that function; the procedure is then known as gradient ascent.

Mean squared error

In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value. MSE is a risk function, corresponding to the expected value of the squared error loss. The fact that MSE is almost always strictly positive (and not zero) is because of randomness or because the estimator does not account for information that could produce a more accurate estimate.

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Gradient Descent for Linear MSE CS-433: Machine learning

Explores Gradient Descent for Linear MSE in machine learning, covering computation, complexity, variants, Stochastic Gradient Descent, penalty functions, implementation issues, and non-convex optimization.

Logistic Regression: Vegetation Prediction MGT-448: Statistical inference and machine learning

Explores logistic regression for predicting vegetation proportions in the Amazon region through remote sensing data analysis.

Optimization: Gradient Descent CS-433: Machine learning

Covers the Gradient Descent algorithm for optimizing cost functions in machine learning.

Document Analysis: Topic Modeling DH-406: Machine learning for DH

Explores document analysis, topic modeling, and generative models for data generation in machine learning.

Optimization in Machine Learning: Gradient Descent BIO-322: Introduction to machine learning for bioengineers

Covers optimization in machine learning, focusing on gradient descent for linear and logistic regression, stochastic gradient descent, and practical considerations.