Publications related to Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Deep Learning Theory Through the Lens of Diagonal Linear Networks

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Explainable Face Verification via Feature-Guided Gradient Backpropagation

Touradj Ebrahimi, Yuhang Lu, Zewei Xu

Recent years have witnessed significant advance- ment in face recognition (FR) techniques, with their applications widely spread in people’s lives and security-sensitive areas. There is a growing need for reliable interpretations of decisions of such syste ...

2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Deep Learning Generalization with Limited and Noisy Labels

Mahsa Forouzesh

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

EPFL2023

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Giancarlo Ferrari Trecate, Luca Furieri, Clara Lucía Galimberti, Liang Xu

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...

2023

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks

Elisabetta Cornacchia

This thesis focuses on two selected learning problems: 1) statistical inference on graphs models, and, 2) gradient descent on neural networks, with the common objective of defining and analysing the measures that characterize the fundamental limits.In the ...

EPFL2023

Polynomial-time universality and limitations of deep learning

Emmanuel Abbé

The goal of this paper is to characterize function distributions that general neural networks trained by descent algorithms (GD/SGD), can or cannot learn in polytime. The results are: (1) The paradigm of general neural networks trained by SGD is poly-time ...

WILEY2023

Predicting nonlinear optical scattering with physics-driven neural networks

Demetri Psaltis, Carlo Gigli, Ahmed Ayoub

Deep neural networks trained on physical losses are emerging as promising surrogates for nonlinear numerical solvers. These tools can predict solutions to Maxwell's equations and compute gradients of output fields with respect to the material and geometric ...

AIP Publishing2023

Transformer Models for Vision

Jean-Baptiste Francis Marie Juliette Cordonnier

The recent developments of deep learning cover a wide variety of tasks such as image classification, text translation, playing go, and folding proteins.All these successful methods depend on a gradient-based learning algorithm to train a model on massive a ...

EPFL2023

Communication-efficient distributed training of machine learning models

Thijs Vogels

In this thesis, we explore techniques for addressing the communication bottleneck in data-parallel distributed training of deep learning models. We investigate algorithms that either reduce the size of the messages that are exchanged between workers, or th ...

EPFL2023

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Graph Chatbot

Chat with Graph Search

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Explainable Face Verification via Feature-Guided Gradient Backpropagation

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Deep Learning Generalization with Limited and Noisy Labels

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks

Polynomial-time universality and limitations of deep learning

Predicting nonlinear optical scattering with physics-driven neural networks

Transformer Models for Vision

Communication-efficient distributed training of machine learning models

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Deep Learning Generalization with Limited and Noisy Labels

Transformer Models for Vision

Explainable Face Verification via Feature-Guided Gradient Backpropagation

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks

Polynomial-time universality and limitations of deep learning

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Communication-efficient distributed training of machine learning models

Predicting nonlinear optical scattering with physics-driven neural networks