Concept

Vanishing gradient problem

Publications associées (126)

Subquadratic Overparameterization for Shallow Neural Networks

Volkan Cevher, Thomas Michaelsen Pethick, Armin Eftekhari, Chaehwan Song

Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various i ...

2021

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

The way our brain learns to disentangle complex signals into unambiguous concepts is fascinating but remains largely unknown. There is evidence, however, that hierarchical neural representations play a key role in the cortex. This thesis investigates biolo ...

EPFL2021

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Maksym Andriushchenko

Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: tr ...

2021

Tracking and Relative Localization of Drone Swarms With a Vision-Based Headset

Dario Floreano, Fabrizio Schiano, Maxim Pavliv, Giuseppe Loianno

We address the detection, tracking, and relative localization of the agents of a drone swarm from a human perspective using a headset equipped with a single camera and an Inertial Measurement Unit (IMU). We train and deploy a deep neural network detector o ...

2021

Crowding and the importance of grouping and segmentation processes in human vision

Alban Bornet

Human vision has evolved to make sense of a world in which elements almost never appear in isolation. Surprisingly, the recognition of an element in a visual scene is strongly limited by the presence of other nearby elements, a phenomenon known as visual c ...

EPFL2021

A Fundamental Limit of Distributed Hypothesis Testing Under Memoryless Quantization

Ali H. Sayed, Emre Telatar, Mert Kayaalp, Yunus Inan

We consider a distributed binary hypothesis testing setup where multiple nodes send quantized information to a central processor, which is oblivious to the nodes’ statistics. We study the regime where the type-II error decays exponentially and the type-I e ...

2021

Landscape and training regimes in deep learning

Matthieu Wyart, Mario Geiger, Leonardo Petrini

Deep learning algorithms are responsible for a technological revolution in a variety oftasks including image recognition or Go playing. Yet, why they work is not understood.Ultimately, they manage to classify data lying in high dimension – a feat generical ...

2021

A unified framework for Hamiltonian deep neural networks

Giancarlo Ferrari Trecate, Clara Lucía Galimberti, Liang Xu

Training deep neural networks (DNNs) can be difficult due to the occurrence of vanishing/explod- ing gradients during weight optimization. To avoid this problem, we propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The t ...

2021

Multi-Gradient Descent for Multi-Objective Recommender Systems

Boi Faltings, Claudiu-Cristian Musat, Diego Matteo Antognini

Recommender systems need to mirror the complexity of the environment they are applied in. The more we know about what might benefit the user, the more objectives the recommender system has. In addition there may be multiple stakeholders - sellers, buyers, ...

2020

ESSOP: Efficient and Scalable Stochastic Outer Product Architecture for Deep Learning

Irem Boybat Kara, Evangelos Eleftheriou, Abu Sebastian

Deep neural networks (DNNs) have surpassed human-level accuracy in a variety of cognitive tasks but at the cost of significant memory/time requirements in DNN training. This limits their deployment in energy and memory limited applications that require rea ...

IEEE2020

Graph Chatbot

Chattez avec Graph Search

Posez n’importe quelle question sur les cours, conférences, exercices, recherches, actualités, etc. de l’EPFL ou essayez les exemples de questions ci-dessous.

AVERTISSEMENT : Le chatbot Graph n'est pas programmé pour fournir des réponses explicites ou catégoriques à vos questions. Il transforme plutôt vos questions en demandes API qui sont distribuées aux différents services informatiques officiellement administrés par l'EPFL. Son but est uniquement de collecter et de recommander des références pertinentes à des contenus que vous pouvez explorer pour vous aider à répondre à vos questions.

Connectez-vous pour utiliser Chat avec Graph Search