Publications related to Second-order guarantees in centralized, federated and decentralized nonconvex optimization

Deep Learning Theory Through the Lens of Diagonal Linear Networks

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Explainable Face Verification via Feature-Guided Gradient Backpropagation

Touradj Ebrahimi, Yuhang Lu, Zewei Xu

Recent years have witnessed significant advance- ment in face recognition (FR) techniques, with their applications widely spread in people’s lives and security-sensitive areas. There is a growing need for reliable interpretations of decisions of such syste ...

2024

Transformer Models for Vision

Jean-Baptiste Francis Marie Juliette Cordonnier

The recent developments of deep learning cover a wide variety of tasks such as image classification, text translation, playing go, and folding proteins.All these successful methods depend on a gradient-based learning algorithm to train a model on massive a ...

EPFL2023

Probabilistic methods for neural combinatorial optimization

Nikolaos Karalias

The monumental progress in the development of machine learning models has led to a plethora of applications with transformative effects in engineering and science. This has also turned the attention of the research community towards the pursuit of construc ...

EPFL2023

Deep Learning Generalization with Limited and Noisy Labels

Mahsa Forouzesh

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

EPFL2023

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective

Daniel Kuhn, Yves Rychener, Tobias Sutter

We develop a principled approach to end-to-end learning in stochastic optimization. First, we show that the standard end-to-end learning algorithm admits a Bayesian interpretation and trains a posterior Bayes action map. Building on the insights of this an ...

2023

Predicting in Uncertain Environments: Methods for Robust Machine Learning

Paul Thierry Yves Rolland

One of the main goal of Artificial Intelligence is to develop models capable of providing valuable predictions in real-world environments. In particular, Machine Learning (ML) seeks to design such models by learning from examples coming from this same envi ...

EPFL2022

Towards Verifiable, Generalizable and Efficient Robust Deep Neural Networks.

Chen Liu

In the last decade, deep neural networks have achieved tremendous success in many fields of machine learning.However, they are shown vulnerable against adversarial attacks: well-designed, yet imperceptible, perturbations can make the state-of-the-art deep ...

EPFL2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Second-order guarantees in centralized, federated and decentralized nonconvex optimization

Graph Chatbot

Chat with Graph Search

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Explainable Face Verification via Feature-Guided Gradient Backpropagation

Transformer Models for Vision

Probabilistic methods for neural combinatorial optimization

Deep Learning Generalization with Limited and Noisy Labels

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective

Predicting in Uncertain Environments: Methods for Robust Machine Learning

Towards Verifiable, Generalizable and Efficient Robust Deep Neural Networks.

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Probabilistic methods for neural combinatorial optimization

Deep Learning Generalization with Limited and Noisy Labels

Towards Verifiable, Generalizable and Efficient Robust Deep Neural Networks.

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Explainable Face Verification via Feature-Guided Gradient Backpropagation

Transformer Models for Vision

Predicting in Uncertain Environments: Methods for Robust Machine Learning

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective