Highway network

In machine learning, the Highway Network was the first working very deep feedforward neural network with hundreds of layers, much deeper than previous artificial neural networks. It uses skip connections modulated by learned gating mechanisms to regulate information flow, inspired by Long Short-Term Memory (LSTM) recurrent neural networks. The advantage of a Highway Network over the common deep neural networks is that it solves or partially prevents the vanishing gradient problem, thus leading to easier to optimize neural networks. The gating mechanisms facilitate information flow across many layers ("information highways"). Highway Networks have been used as part of text sequence labeling and speech recognition tasks. An open-gated or gateless Highway Network variant called Residual neural network was used to win the ImageNet 2015 competition. This has become the most cited neural network of the 21st century. The model has two gates in addition to the H(WH, x) gate: the transform gate T(WT, x) and the carry gate C(WC, x). Those two last gates are non-linear transfer functions (by convention Sigmoid function). The H(WH, x) function can be any desired transfer function. The carry gate is defined as C(WC, x) = 1 - T(WT, x). While the transform gate is just a gate with a sigmoid transfer function.

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Publications associées (52)

Personnes associées (13)

Unités associées (16)

Concepts associés (4)

Cours associés (2)

Séances de cours associées (34)

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Scott William Pesme

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

, ,

The integration of smart thermostats in home automation systems has created an opportunity to optimize space heating and cooling through the use of machine learning, for example for thermal model identification. Nonetheless, its full potential remains unta ...

New York2023

Universal Approximation Property of Hamiltonian Deep Neural Networks

Giancarlo Ferrari Trecate, Muhammad Zakwan

This letter investigates the universal approximation capabilities of Hamiltonian Deep Neural Networks (HDNNs) that arise from the discretization of Hamiltonian Neural Ordinary Differential Equations. Recently, it has been shown that HDNNs enjoy, by design, ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2023

Vanishing gradient problem

In machine learning, the vanishing gradient problem is encountered when training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, during each iteration of training each of the neural networks weights receives an update proportional to the partial derivative of the error function with respect to the current weight. The problem is that in some cases, the gradient will be vanishingly small, effectively preventing the weight from changing its value.

Long short-term memory

Long short-term memory (LSTM) network is a recurrent neural network (RNN), aimed to deal with the vanishing gradient problem present in traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, hidden Markov models and other sequence learning methods. It aims to provide a short-term memory for RNN that can last thousands of timesteps, thus "long short-term memory".

Reconnaissance automatique de la parole

vignette|droite|upright=1.4|La reconnaissance vocale est habituellement traitée dans le middleware ; les résultats sont transmis aux applications utilisatrices. La reconnaissance automatique de la parole (souvent improprement appelée reconnaissance vocale) est une technique informatique qui permet d'analyser la voix humaine captée au moyen d'un microphone pour la transcrire sous la forme d'un texte exploitable par une machine.

FIN-407: Machine learning in finance

This course aims to give an introduction to the application of machine learning to finance. These techniques gained popularity due to the limitations of traditional financial econometrics methods tack

CIVIL-459: Deep learning for autonomous vehicles

Deep Learning (DL) is the subset of Machine learning reshaping the future of transportation and mobility. In this class, we will show how DL can be used to teach autonomous vehicles to detect objects,

Se concentre sur l'estimation de la pose articulaire de la balle et de la poche à l'aide d'un champ magnétique dans une conception anthropomorphe avec muscles artificiels et tendons.

Explore le mécanisme d'attention dans la traduction automatique, en s'attaquant au problème du goulot d'étranglement et en améliorant considérablement les performances NMT.

Explore le but et le processus de normalisation par lots dans les réseaux neuronaux profonds, en soulignant son importance dans la stabilisation de l'entrée moyenne et la résolution du problème du gradient de fuite.