Adaptive Optimization Methods: Theory and Applications

This lecture covers stochastic adaptive first-order methods that converge without knowing the smoothness constant by utilizing information from stochastic gradients. It introduces variable metric stochastic gradient descent algorithms and adaptive gradient methods that locally adapt by setting the Hessian matrix based on past stochastic gradient information. The lecture also discusses AdaGrad, AcceleGrad, RMSProp, and ADAM, highlighting their properties and convergence rates. It compares various adaptive algorithms, including their performance in optimization tasks and generalization capabilities. The implications of implicit regularization in adaptive methods and the generalization performance of adaptive learning methods are also explored.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (36)

Related concepts (33)

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data).

Multi-objective optimization or Pareto optimization (also known as multi-objective programming, vector optimization, multicriteria optimization, or multiattribute optimization) is an area of multiple-criteria decision making that is concerned with mathematical optimization problems involving more than one objective function to be optimized simultaneously. Multi-objective is a type of vector optimization that has been applied in many fields of science, including engineering, economics and logistics where optimal decisions need to be taken in the presence of trade-offs between two or more conflicting objectives.

In computer science and operations research, the ant colony optimization algorithm (ACO) is a probabilistic technique for solving computational problems which can be reduced to finding good paths through graphs. Artificial ants stand for multi-agent methods inspired by the behavior of real ants. The pheromone-based communication of biological ants is often the predominant paradigm used. Combinations of artificial ants and local search algorithms have become a method of choice for numerous optimization tasks involving some sort of graph, e.

In machine learning, hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned. The same kind of machine learning model can require different constraints, weights or learning rates to generalize different data patterns.

In mathematics and computer science, an algorithm (ˈælɡərɪðəm) is a finite sequence of rigorous instructions, typically used to solve a class of specific problems or to perform a computation. Algorithms are used as specifications for performing calculations and data processing. More advanced algorithms can use conditionals to divert the code execution through various routes (referred to as automated decision-making) and deduce valid inferences (referred to as automated reasoning), achieving automation eventually.

Adaptive Gradient Methods: Theory and Applications EE-556: Mathematics of data: from theory to computation

Explores adaptive gradient methods, their properties, convergence, and comparison with traditional optimization algorithms.

Double Descent Curves: Overparametrization EE-556: Mathematics of data: from theory to computation

Explores double descent curves and overparametrization in machine learning models, highlighting the risks and benefits.

Multilayer Neural Networks: Deep Learning PHYS-467: Machine learning for physicists

Covers the fundamentals of multilayer neural networks and deep learning.

Optimality of Convergence Rates: Accelerated Gradient Descent EE-556: Mathematics of data: from theory to computation

Explores the optimality of convergence rates in convex optimization, focusing on accelerated gradient descent and adaptive methods.

Optimality of Convergence Rates: Accelerated/Stochastic Gradient Descent EE-556: Mathematics of data: from theory to computation

Covers the optimality of convergence rates in accelerated and stochastic gradient descent methods for non-convex optimization problems.