Publication

A Theory of Finite-Width Neural Networks: Generalization, Scaling Laws, and the Loss Landscape

Related publications (57)

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Continuous-variable neural network quantum states and the quantum rotor model

Giuseppe Carleo

We initiate the study of neural network quantum state algorithms for analyzing continuous-variable quantum systems in which the quantum degrees of freedom correspond to coordinates on a smooth manifold. A simple family of continuous-variable trial wavefunc ...

SPRINGERNATURE2023

We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on the learned parameters and predictors. We first describe a general class of s ...

AMER INST MATHEMATICAL SCIENCES-AIMS2023

Learning curves for the multi-class teacher-student perceptron

Lenka Zdeborová, Elisabetta Cornacchia, Bruno Loureiro, Bruno Loureiro, Francesca Mignacco

One of the most classical results in high-dimensional learning theory provides a closed-form expression for the generalisation error of binary classification with a single-layer teacher-student perceptron on i.i.d. Gaussian inputs. Both Bayes-optimal (BO) ...

IOP Publishing Ltd2023

Universal Approximation Property of Hamiltonian Deep Neural Networks

Giancarlo Ferrari Trecate, Muhammad Zakwan

This letter investigates the universal approximation capabilities of Hamiltonian Deep Neural Networks (HDNNs) that arise from the discretization of Hamiltonian Neural Ordinary Differential Equations. Recently, it has been shown that HDNNs enjoy, by design, ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2023

Sharp asymptotics on the compression of two-layer neural networks

Marco Mondelli

In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M < N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimiz ...

IEEE2022

Theory of Deep Learning: Neural Tangent Kernel and Beyond

Arthur Ulysse Jacot-Guillarmod

In the recent years, Deep Neural Networks (DNNs) have managed to succeed at tasks that previously appeared impossible, such as human-level object recognition, text synthesis, translation, playing games and many more. In spite of these major achievements, o ...

EPFL2022

Learning of Continuous and Piecewise-Linear Functions With Hessian Total-Variation Regularization

Michaël Unser, Shayan Aziznejad, Joaquim Gonçalves Garcia Barreto Campos

We develop a novel 2D functional learning framework that employs a sparsity-promoting regularization based on second-order derivatives. Motivated by the nature of the regularizer, we restrict the search space to the span of piecewise-linear box splines shi ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Proximal Point Imitation Learning

Volkan Cevher, Luca Viano, Igor Krawczuk, Angeliki Kamoutsi

This work develops new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning (IL) with linear function approximation without restrictive coherence assumptions. We begin with the minimax formulation of the problem and then o ...

2022

, ,

The "quasi-constant " smooth overlap of atomic position and atom-centered symmetry function fingerprint manifolds recently discovered by Parsaeifard and Goedecker [J. Chem. Phys. 156, 034302 (2022)] are closely related to the degenerate pairs of configurat ...

AIP Publishing2022

An Analysis of Super-Net Heuristics in Weight-Sharing NAS

Mathieu Salzmann, Kaicheng Yu

Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since he ...

IEEE COMPUTER SOC2022