Publications related to Disparity Between Batches as a Signal for Early Stopping

Deep Learning Theory Through the Lens of Diagonal Linear Networks

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Reduced Training Data for Laser Ultrasound Signal Interpretation by Neural Networks

Romain Christophe Rémy Fleury, Janez Rus

The performance of machine learning algorithms is conditioned by the availability of training datasets, which is especially true for the field of nondestructive evaluation. Here we propose one reconfigurable specimen instead of numerous reference specimens ...

2024

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Brookline2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Deep Learning Generalization with Limited and Noisy Labels

Mahsa Forouzesh

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

EPFL2023

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Michaël Unser

The minimization of a data-fidelity term and an additive regularization functional gives rise to a powerful framework for supervised learning. In this paper, we present a unifying regularization functional that depends on an operator L\documentclass[12pt]{ ...

New York2023

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Giancarlo Ferrari Trecate, Luca Furieri, Clara Lucía Galimberti, Liang Xu

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...

2023

Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations

Leonardo Petrini

Artificial intelligence, particularly the subfield of machine learning, has seen a paradigm shift towards data-driven models that learn from and adapt to data. This has resulted in unprecedented advancements in various domains such as natural language proc ...

EPFL2023

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective

Daniel Kuhn, Yves Rychener, Tobias Sutter

We develop a principled approach to end-to-end learning in stochastic optimization. First, we show that the standard end-to-end learning algorithm admits a Bayesian interpretation and trains a posterior Bayes action map. Building on the insights of this an ...

2023

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks

Elisabetta Cornacchia

This thesis focuses on two selected learning problems: 1) statistical inference on graphs models, and, 2) gradient descent on neural networks, with the common objective of defining and analysing the measures that characterize the fundamental limits.In the ...

EPFL2023

Disparity Between Batches as a Signal for Early Stopping

Graph Chatbot

Chat with Graph Search

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Reduced Training Data for Laser Ultrasound Signal Interpretation by Neural Networks

On the Generalization of Stochastic Gradient Descent with Momentum

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Deep Learning Generalization with Limited and Noisy Labels

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Deep Learning Generalization with Limited and Noisy Labels

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks

On the Generalization of Stochastic Gradient Descent with Momentum

Reduced Training Data for Laser Ultrasound Signal Interpretation by Neural Networks

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective