Publications related to Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Deep Learning Theory Through the Lens of Diagonal Linear Networks

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Generalization of Scaled Deep ResNets in the Mean-Field Regime

Volkan Cevher, Grigorios Chrysos, Fanghui Liu

Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate scaled ResNet in the limit of infinitely deep and wide neural networks, of wh ...

2024

Task-driven neural network models predict neural dynamics of proprioception: Experimental data, activations and predictions of neural network models

Alexander Mathis, Alberto Silvio Chiappa, Alessandro Marin Vargas, Axel Bisi

Here we provide the neural data, activation and predictions for the best models and result dataframes of our article "Task-driven neural network models predict neural dynamics of proprioception". It contains the behavioral and neural experimental data (cu ...

EPFL Infoscience2024

Gibbs sampling the posterior of neural networks

Lenka Zdeborová, Giovanni Piccioli, Emanuele Troiani

In this paper, we study sampling from a posterior derived from a neural network. We propose a new probabilistic model consisting of adding noise at every pre- and post-activation in the network, arguing that the resulting posterior can be sampled using an ...

Bristol2024

Performing and Detecting Backdoor Attacks on Face Recognition Algorithms

Alexander Carl Unnervik

The field of biometrics, and especially face recognition, has seen a wide-spread adoption the last few years, from access control on personal devices such as phones and laptops, to automated border controls such as in airports. The stakes are increasingly ...

EPFL2024

Sparse autoregressive neural networks for classical spin systems

Giuseppe Carleo, Dian Wu, Indaco Biazzo

Efficient sampling and approximation of Boltzmann distributions involving large sets of binary variables, or spins, are pivotal in diverse scientific fields even beyond physics. Recent advances in generative neural networks have significantly impacted this ...

Iop Publishing Ltd2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Detecting Road Obstacles by Erasing Them

Pascal Fua, Mathieu Salzmann, Krzysztof Maciej Lis, Sina Honari

Vehicles can encounter a myriad of obstacles on the road, and it is impossible to record them all beforehand to train a detector. Instead, we select image patches and inpaint them with the surrounding road texture, which tends to remove obstacles from thos ...

2024

Deep Learning Generalization with Limited and Noisy Labels

Mahsa Forouzesh

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

EPFL2023

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Giancarlo Ferrari Trecate, Luca Furieri, Clara Lucía Galimberti, Liang Xu

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...

2023

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Graph Chatbot

Chat with Graph Search

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Generalization of Scaled Deep ResNets in the Mean-Field Regime

Task-driven neural network models predict neural dynamics of proprioception: Experimental data, activations and predictions of neural network models

Gibbs sampling the posterior of neural networks

Performing and Detecting Backdoor Attacks on Face Recognition Algorithms

Sparse autoregressive neural networks for classical spin systems

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Detecting Road Obstacles by Erasing Them

Deep Learning Generalization with Limited and Noisy Labels

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Generalization of Scaled Deep ResNets in the Mean-Field Regime

Gibbs sampling the posterior of neural networks

Performing and Detecting Backdoor Attacks on Face Recognition Algorithms

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Detecting Road Obstacles by Erasing Them

Deep Learning Generalization with Limited and Noisy Labels

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Task-driven neural network models predict neural dynamics of proprioception: Experimental data, activations and predictions of neural network models

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Sparse autoregressive neural networks for classical spin systems