Publications associées à Vanishing gradient problem

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in te ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach

Giancarlo Ferrari Trecate, Luca Furieri, Muhammad Zakwan, Clara Lucía Galimberti

Large-scale cyber-physical systems require that control policies are distributed, that is, that they only rely on local real-time measurements and communication with neighboring agents. Optimal Distributed Control (ODC) problems are, however, highly intrac ...

2022

Robust Binary Models by Pruning Randomly-initialized Networks

Sabine Süsstrunk, Mathieu Salzmann, Chen Liu, Ziqi Zhao

Robustness to adversarial attacks was shown to require a larger model capacity, and thus a larger memory footprint. In this paper, we introduce an approach to obtain robust yet compact models by pruning randomly-initialized binary networks. Unlike adversar ...

2022

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generaliza ...

2022

Language Transformers for Remote Sensing Visual Question Answering

Devis Tuia, Sylvain Lobry, Christel Marie Tartini-Chappuis, Vincent Alexandre Mendez

Remote sensing visual question answering (RSVQA) opens new avenues to promote the use of satellites data, by interfacing satellite image analysis with natural language processing. Capitalizing on the remarkable advances in natural language processing and c ...

2022

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Loucas Pillaud-Vivien

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, ...

2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Approximating Relu Networks By Single-Spike Computation

Wulfram Gerstner, Stanislaw Andrzej Wozniak, Ana Stanojevic, Evangelos Eleftheriou, Giovanni Cherubini, Angeliki Pantazi

Developing energy-saving neural network models is a topic of rapidly increasing interest in the artificial intelligence community. Spiking neural networks (SNNs) are biologically inspired models that strive to leverage the energy efficiency stemming from a ...

IEEE2022

Chattez avec Graph Search

Posez n’importe quelle question sur les cours, conférences, exercices, recherches, actualités, etc. de l’EPFL ou essayez les exemples de questions ci-dessous.

AVERTISSEMENT : Le chatbot Graph n'est pas programmé pour fournir des réponses explicites ou catégoriques à vos questions. Il transforme plutôt vos questions en demandes API qui sont distribuées aux différents services informatiques officiellement administrés par l'EPFL. Son but est uniquement de collecter et de recommander des références pertinentes à des contenus que vous pouvez explorer pour vous aider à répondre à vos questions.

Vanishing gradient problem

ADAGRAD Avoids Saddle Points

ADAGRAD Avoids Saddle Points

Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach

Robust Binary Models by Pruning Randomly-initialized Networks

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Generalization Properties of NAS under Activation and Skip Connection Search

Language Transformers for Remote Sensing Visual Question Answering

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Approximating Relu Networks By Single-Spike Computation

Graph Chatbot

Chattez avec Graph Search

Robust Binary Models by Pruning Randomly-initialized Networks

ADAGRAD Avoids Saddle Points

ADAGRAD Avoids Saddle Points

Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Approximating Relu Networks By Single-Spike Computation

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Language Transformers for Remote Sensing Visual Question Answering