Optimization Methods in Machine Learning

This lecture delves into optimization methods in machine learning, focusing on gradient descent and stochastic gradient descent. The instructor explains the importance of understanding gradients, costs, and computational efforts in optimization. The lecture covers topics such as full gradient computation, individual element gradients, subgradients, and convexity. Additionally, the instructor discusses the concept of subgradients for non-differentiable functions and the use of penalties to encourage sparsity in models. The lecture also touches on optimality conditions, step size selection, and preconditioning to improve optimization performance.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (774)

Related concepts (188)

CS-433: Machine learning

Machine learning methods are becoming increasingly central in many sciences and applications. In this course, fundamental principles and methods of machine learning will be introduced, analyzed and pr

Convex optimization

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Stochastic gradient descent

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data).

Constrained optimization

In mathematical optimization, constrained optimization (in some contexts called constraint optimization) is the process of optimizing an objective function with respect to some variables in the presence of constraints on those variables. The objective function is either a cost function or energy function, which is to be minimized, or a reward function or utility function, which is to be maximized.

Mathematical optimization

Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries.

Optimization problem

In mathematics, computer science and economics, an optimization problem is the problem of finding the best solution from all feasible solutions. Optimization problems can be divided into two categories, depending on whether the variables are continuous or discrete: An optimization problem with discrete variables is known as a discrete optimization, in which an object such as an integer, permutation or graph must be found from a countable set.

Gradient Descent for Linear MSE CS-433: Machine learning

Explores Gradient Descent for Linear MSE in machine learning, covering computation, complexity, variants, Stochastic Gradient Descent, penalty functions, implementation issues, and non-convex optimization.

Optimization Methods: Theory Discussion ME-454: Modelling and optimization of energy systems

Explores optimization methods, including unconstrained problems, linear programming, and heuristic approaches.

Optimisation in Energy Systems ME-454: Modelling and optimization of energy systems

Explores optimization in energy system modeling, covering decision variables, objective functions, and different strategies with their pros and cons.

Optimization Problems: Path Finding and Portfolio Allocation MGT-483: Optimal decision making

Covers optimization problems in path finding and portfolio allocation.

Thermodynamic Properties: Equations and Models ME-454: Modelling and optimization of energy systems

Explains thermodynamic properties, equations of state, and mixture rules for energy systems modeling.