Publications related to Softmax function

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

On the number of regions of piecewise linear neural networks

Michaël Unser, Alexis Marie Frederic Goujon

Many feedforward neural networks (NNs) generate continuous and piecewise-linear (CPWL) mappings. Specifically, they partition the input domain into regions on which the mapping is affine. The number of these so-called linear regions offers a natural metric ...

2024

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Scott William Pesme

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Towards Trustworthy Deep Learning for Image Reconstruction

Alexis Marie Frederic Goujon

The remarkable ability of deep learning (DL) models to approximate high-dimensional functions from samples has sparked a revolution across numerous scientific and industrial domains that cannot be overemphasized. In sensitive applications, the good perform ...

EPFL2024

Benign Overfitting in Deep Neural Networks under Lazy Training

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayesoptimal test error for classification while obtaining (nearly) zero-trai ...

2023

A Theory of Finite-Width Neural Networks: Generalization, Scaling Laws, and the Loss Landscape

Berfin Simsek

Deep learning has achieved remarkable success in various challenging tasks such as generating images from natural language or engaging in lengthy conversations with humans.The success in practice stems from the ability to successfully train massive neural ...

EPFL2023

Inverse Reinforcement Learning of Pedestrian-Robot Coordination

Aude Billard, David Julian Gonon

We apply inverse reinforcement learning (IRL) with a novel cost feature to the problem of robot navigation in human crowds. Consistent with prior empirical work on pedestrian behavior, the feature anticipates collisions between agents. We efficiently learn ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2023

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Michaël Unser

The minimization of a data-fidelity term and an additive regularization functional gives rise to a powerful framework for supervised learning. In this paper, we present a unifying regularization functional that depends on an operator L\documentclass[12pt]{ ...

New York2023

Bayes-optimal Learning of Deep Random Networks of Extensive-width

Florent Gérard Krzakala, Lenka Zdeborová, Hugo Chao Cui

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width ...

2023

Penalising the biases in norm regularisation enforces sparsity

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier

Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between parameters' norm and obtained estimators theoretically remains misunderstood. For one hidden ReLU layer networks ...

2023

Softmax function

Graph Chatbot

Chat with Graph Search

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

On the number of regions of piecewise linear neural networks

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Towards Trustworthy Deep Learning for Image Reconstruction

Benign Overfitting in Deep Neural Networks under Lazy Training

A Theory of Finite-Width Neural Networks: Generalization, Scaling Laws, and the Loss Landscape

Inverse Reinforcement Learning of Pedestrian-Robot Coordination

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Bayes-optimal Learning of Deep Random Networks of Extensive-width

Penalising the biases in norm regularisation enforces sparsity

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

On the number of regions of piecewise linear neural networks

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Towards Trustworthy Deep Learning for Image Reconstruction

Bayes-optimal Learning of Deep Random Networks of Extensive-width

Benign Overfitting in Deep Neural Networks under Lazy Training

Inverse Reinforcement Learning of Pedestrian-Robot Coordination

A Theory of Finite-Width Neural Networks: Generalization, Scaling Laws, and the Loss Landscape

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Penalising the biases in norm regularisation enforces sparsity