Publications by Stefano Spigler

How isotropic kernels perform on simple invariants

We investigate how the training curve of isotropic kernel methods depends on the symmetry of the task to be learned, in several settings. (i) We consider a regression task, where the target function is a Gaussian random field that depends only on d(paralle ...

IOP PUBLISHING LTD2021

Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm

Matthieu Wyart, Mario Geiger, Stefano Spigler

How many training data are needed to learn a supervised task? It is often observed that the generalization error decreases as n(-beta) where n is the number of training examples and beta is an exponent that depends on both data and algorithm. In this work ...

IOP PUBLISHING LTD2020

Disentangling feature and lazy training in deep neural networks

Matthieu Wyart, Mario Geiger, Stefano Spigler, Arthur Ulysse Jacot-Guillarmod, Arthur Jacot

Two distinct limits for deep learning have been derived as the network width h -> infinity, depending on how the weights of the last layer scale with h. In the neural tangent Kernel (NTK) limit, the dynamics becomes linear in the weights and is described b ...

IOP PUBLISHING LTD2020

Scaling description of generalization with number of parameters in deep learning

Matthieu Wyart, Clément Hongler, Mario Geiger, Stefano Spigler, Levent Dogus Sagun, Franck Raymond Gabriel, Arthur Ulysse Jacot-Guillarmod, Arthur Jacot

Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsitybased arguments would suggest th ...

IOP PUBLISHING LTD2020

A jamming transition from under- to over-parametrization affects generalization in deep learning

Matthieu Wyart, Stefano Spigler, Levent Dogus Sagun

In this paper we first recall the recent result that in deep networks a phase transition, analogous to the jamming transition of granular media, delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. The analysis leading ...

IOP PUBLISHING LTD2019

Comparing dynamics: deep neural networks versus glassy systems

Matthieu Wyart, Mario Geiger, Marco Baity-Jesi, Stefano Spigler, Levent Dogus Sagun

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and ( ...

IOP PUBLISHING LTD2019

Jamming transition as a paradigm to understand the loss landscape of deep neural networks

Matthieu Wyart, Mario Geiger, Marco Baity-Jesi, Stefano Spigler, Levent Dogus Sagun

Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understandin ...

AMER PHYSICAL SOC2019

Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Rainer Wesche, Marco Baity-Jesi, Mario Geiger, Stefano Spigler, Levent Dogus Sagun

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are the complexity of the loss-landscape and of the dynamics within it, and to wh ...

2018