Publications related to Generalization error

Semantic Perturbations with Normalizing Flows for Improved Generalization

Martin Jaggi, Sebastian Urban Stich, Tatjana Chavdarova

Data augmentation is a widely adopted technique for avoiding overfitting when training deep neural networks. However, this approach requires domain-specific knowledge and is often limited to a fixed set of hard-coded transformations. Recently, several work ...

IEEE2021

KPC: Learning-Based Model Predictive Control with Deterministic Guarantees

Colin Neil Jones, Yuning Jiang, Paul Scharnhorst, Emilio Maddalena

We propose Kernel Predictive Control (KPC), a learning-based predictive control strategy that enjoys deterministic guarantees of safety. Noise-corrupted samples of the unknown system dynamics are used to learn several models through the formalism of non-pa ...

2021

Consensus Control for Decentralized Deep Learning

Martin Jaggi, Sebastian Urban Stich, Anastasiia Koloskova, Tao Lin, Lingjing Kong

Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers fro ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

Structure of multicorrelation sequences with integer part polynomial iterates along primes

Florian Karl Richter

Let T be a measure-preserving Zℓ-action on the probability space (X,B,μ), let q1,…,qm:R→Rℓ be vector polynomials, and let f0,…,fm∈L∞⁡(X). For any ϵ>0 and multicorrelation sequences of the form α⁡(n)=∫Xf0⋅T⌊q1⁡(n)⌋⁢f1⋯T⌊qm⁡(n)⌋⁢fmd⁢μ we show that there exis ...

2021

Subquadratic Overparameterization for Shallow Neural Networks

Volkan Cevher, Thomas Michaelsen Pethick, Armin Eftekhari, Chaehwan Song

Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various i ...

2021

Robust Generalization via $\alpha$-Mutual Information

Michael Christoph Gastpar, Amedeo Roberto Esposito, Ibrahim Issa

The aim of this work is to provide bounds connecting two probability measures of the same event using Rényi

\alpha

-Divergences and Sibson’s

\alpha

-Mutual Information, a generalization of respectively the Kullback-Leibler Divergence and Shannon’s Mutual ...

ETHZ2020

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

Konstantinos Pitas

Deep neural networks have been empirically successful in a variety of tasks, however their theoretical understanding is still poor. In particular, modern deep neural networks have many more parameters than training data. Thus, in principle they should over ...

EPFL2020

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

Florent Gérard Krzakala, Lenka Zdeborová

We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer non-linear neural network with random iid inputs. We study the generalization performances of standard classifiers in the high ...

Curran Associates, Inc.2020

Scaling description of generalization with number of parameters in deep learning

Matthieu Wyart, Clément Hongler, Mario Geiger, Stefano Spigler, Levent Dogus Sagun, Franck Raymond Gabriel, Arthur Jacot

Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsitybased arguments would suggest th ...

IOP PUBLISHING LTD2020

AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

Sabine Süsstrunk, Majed El Helou, Frederike Dümbgen

The large capacity of neural networks enables them to learn complex functions. To avoid overfitting, networks however require a lot of training data that can be expensive and time-consuming to collect. A common practical approach to attenuate overfitting i ...

IEEE2020

Generalization error

Graph Chatbot

Chat with Graph Search

Semantic Perturbations with Normalizing Flows for Improved Generalization

KPC: Learning-Based Model Predictive Control with Deterministic Guarantees

Consensus Control for Decentralized Deep Learning

Structure of multicorrelation sequences with integer part polynomial iterates along primes

Subquadratic Overparameterization for Shallow Neural Networks

Robust Generalization via $\alpha$-Mutual Information

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

Scaling description of generalization with number of parameters in deep learning

AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

KPC: Learning-Based Model Predictive Control with Deterministic Guarantees

Structure of multicorrelation sequences with integer part polynomial iterates along primes

Subquadratic Overparameterization for Shallow Neural Networks

Semantic Perturbations with Normalizing Flows for Improved Generalization

Consensus Control for Decentralized Deep Learning

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

Robust Generalization via $\alpha$-Mutual Information

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

AL2: Progressive Activation Loss for Learning General Representations in Classification Neural Networks

Scaling description of generalization with number of parameters in deep learning