Concept
Batch normalization
Publications associées (58)
Deep Learning Theory Through the Lens of Diagonal Linear Networks
In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of -layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...
EPFL2024On the Generalization of Stochastic Gradient Descent with Momentum
Volkan Cevher, Kimon Antonakopoulos
While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...
Brookline2024Deep Learning Generalization with Limited and Noisy Labels
Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...
EPFL2023Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design
Giancarlo Ferrari Trecate, Luca Furieri, Clara Lucía Galimberti, Liang Xu
Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...
2023Benign Overfitting in Deep Neural Networks under Lazy Training
Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu
This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayesoptimal test error for classification while obtaining (nearly) zero-trai ...
2023An exact mapping from ReLU networks to spiking neural networks
Wulfram Gerstner, Stanislaw Andrzej Wozniak, Ana Stanojevic, Giovanni Cherubini, Angeliki Pantazi
Deep spiking neural networks (SNNs) offer the promise of low-power artificial intelligence. However, training deep SNNs from scratch or converting deep artificial neural networks to SNNs without loss of performance has been a challenge. Here we propose an ...
2023Supervised learning and inference of spiking neural networks with temporal coding
The way biological brains carry out advanced yet extremely energy efficient signal processing remains both fascinating and unintelligible. It is known however that at least some areas of the brain perform fast and low-cost processing relying only on a smal ...
EPFL2023