Adaptive Gradient Methods

This lecture covers adaptive gradient methods, including the Variable Metric Gradient Descent Algorithm and AdaGrad. It explains how these methods adapt locally by setting the Hessian matrix based on past gradient information. The lecture delves into the mathematical details of AdaGrad, highlighting its adaptive step-size and coordinate-wise extension. It also discusses the convergence rates for AdaGrad and introduces AcceleGrad, a combination of adaptive and accelerated algorithms. The lecture further explores UniXGrad, an accelerated extra-gradient algorithm for handling constraints, and ExtraNewton, an adaptive Newton's method with fast convergence rates.