Publications related to Towards Understanding Sharpness-Aware Minimization

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Florent Gérard Krzakala, Lenka Zdeborová, Francesca Mignacco

We analyze in a closed form the learning dynamics of stochastic gradient descent (SGD) for a single layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of ...

Curran Associates, Inc.2020

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Lenka Zdeborová

p>We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the overparametrized regime where the layer width m is larger than the input dimension d. We conside ...

Curran Associates, Inc.2020

ESSOP: Efficient and Scalable Stochastic Outer Product Architecture for Deep Learning

Irem Boybat Kara, Evangelos Eleftheriou, Abu Sebastian

Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require rea ...

IEEE2020

Extrapolation for Large-batch Training in Deep Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batc ...

2020

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

Konstantinos Pitas

Deep neural networks have been empirically successful in a variety of tasks, however their theoretical understanding is still poor. In particular, modern deep neural networks have many more parameters than training data. Thus, in principle they should over ...

EPFL2020

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy

We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show that the rate of convergence in all cases consists of two terms: (i) a stocha ...

MICROTOME PUBL2020

Square Attack: a query-efficient black-box adversarial attack via random search

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko, Francesco Croce

We propose the Square Attack, a new score-based black-box

l_2

and

l_\infty

adversarial attack that does not rely on local gradient information and thus is not affected by gradient masking. The Square Attack is based on a randomized search scheme where ...

2020

A jamming transition from under- to over-parametrization affects generalization in deep learning

Matthieu Wyart, Stefano Spigler, Levent Dogus Sagun

In this paper we first recall the recent result that in deep networks a phase transition, analogous to the jamming transition of granular media, delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. The analysis leading ...

IOP PUBLISHING LTD2019

SparseFool: a few pixels make a big difference

Pascal Frossard, Seyed Mohsen Moosavi Dezfooli, Apostolos Modas

Deep Neural Networks have achieved extraordinary results on image classification tasks, but have been shown to be vulnerable to attacks with carefully crafted perturbations of the input data. Although most attacks usually change values of many image's pixe ...

IEEE2019

Biologically plausible deep learning – but how far can we go with shallow networks?

Wulfram Gerstner, Johanni Michael Brea, Bernd Albert Illing

Training deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining s ...

2019

Towards Understanding Sharpness-Aware Minimization

Graph Chatbot

Chat with Graph Search

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

ESSOP: Efficient and Scalable Stochastic Outer Product Architecture for Deep Learning

Extrapolation for Large-batch Training in Deep Learning

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

Square Attack: a query-efficient black-box adversarial attack via random search

A jamming transition from under- to over-parametrization affects generalization in deep learning

SparseFool: a few pixels make a big difference

Biologically plausible deep learning – but how far can we go with shallow networks?

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Square Attack: a query-efficient black-box adversarial attack via random search

A jamming transition from under- to over-parametrization affects generalization in deep learning

SparseFool: a few pixels make a big difference

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

ESSOP: Efficient and Scalable Stochastic Outer Product Architecture for Deep Learning

Extrapolation for Large-batch Training in Deep Learning

Biologically plausible deep learning – but how far can we go with shallow networks?