Êtes-vous un étudiant de l'EPFL à la recherche d'un projet de semestre?
Travaillez avec nous sur des projets en science des données et en visualisation, et déployez votre projet sous forme d'application sur Graph Search.
In this paper, we focus on a theory-practice gap for Adam and its variants (AMSgrad, AdamNC, etc.). In practice, these algorithms are used with a constant first-order moment parameter 1 (typically between 0:9 and 0:99). In theory, regret guarantees for online convex optimization require a rapidly decaying 1 ! 0 schedule. We show that this is an artifact of the standard analysis and propose a novel framework that allows us to derive optimal, data-dependent regret bounds with a constant 1, without further assumptions. We also demonstrate the flexibility of our analysis on a wide range of different algorithms and settings.
Colin Neil Jones, Yuning Jiang, Bratislav Svetozarevic, Wenjie Xu