Redresseur (réseaux neuronaux)

vignette|Graphique de la fonction Unité Linéaire Rectifiée En mathématiques, la fonction Unité Linéaire Rectifiée (ou ReLU pour Rectified Linear Unit) est définie par : f(x)=max(0,x) pour tout réel x Elle est fréquemment utilisée comme fonction d'activation dans le contexte du réseau de neurones artificiels pour sa simplicité de calcul, en particulier de sa dérivée. Variations Un désavantage de la fonction ReLU est que sa dérivée devient nulle lorsque l'entrée est négative ce qui peut empêcher la rétropropagation du gradient. On peut alors introduire une version appelée Leaky ReLU définie par : f(x)=max(\epsilon x, x) pour tout réel x Le paramètre \epsilon est un réel strictement positif et inférieur à 1 \epsilon \in ]0,1[ La dérivée est alors égale à \epsilon lorsque x est strictement négatif, ce qui permet de conserver la mise à jour des poids d'un perceptron utilisant

Theory of Deep Learning: Neural Tangent Kernel and Beyond

Arthur Ulysse Jacot-Guillarmod

In the recent years, Deep Neural Networks (DNNs) have managed to succeed at tasks that previously appeared impossible, such as human-level object recognition, text synthesis, translation, playing games and many more. In spite of these major achievements, our understanding of these models, in particular of what happens during their training, remains very limited. This PhD started with the introduction of the Neural Tangent Kernel (NTK) to describe the evolution of the function represented by the network during training. In the infinite-width limit, i.e. when the number of neurons in the layers of the network grows to infinity, the NTK converges to a deterministic and time-independent limit, leading to a simple yet complete description of the dynamics of infinitely-wide DNNs. This allowed one to give the first general proof of convergence of DNNs to a global minimum, and yielded the first description of the limiting spectrum of the Hessian of the loss surface of DNNs throughout training.More importantly, the NTK plays a crucial role in describing the generalization abilities of DNNs, i.e. the performance of the trained network on unseen data. The NTK analysis uncovered a direct link between the function learned by infinitely wide DNNs and Kernel Ridge Regression predictors, whose generalization properties are studied in this thesis using tools of random matrix theory. Our analysis of KRR reveals the importance of the eigendecomposition of the NTK, which is affected by a number of architectural choices. In very deep networks, an ordered regime and a chaotic regime appear, determined by the choice of non-linearity and the balance between the weights and bias parameters; these two phases are characterized by different speeds of decay of the eigenvalues of the NTK, leading to a tradeoff between convergence speed and generalization. In practical contexts such as Generative Adversarial Networks or Topology Optimization, the network architecture can be chosen to guarantee certain properties of the NTK and its spectrum.These results give an almost complete description DNNs in this infinite-width limit. It is then natural to wonder how it extends to finite-width networks used in practice. In the so-called NTK regime, the discrepancy between finite- and infinite-widths DNNs is mainly a result of the variance w.r.t. to the sampling of the parameters, as shown empirically and mathematically relying on the similarity between DNNs and random feature models.In contrast to the NTK regime, where the NTK remains constant during training, there exist so-called active regimes, where the evolution of the NTK is significant, which appear in a number of settings. One such regime appears in Deep Linear Networks with a very small initialization, where the training dynamics approach a sequence of saddle-points, representing linear maps of increasing rank, leading to a low-rank bias which is absent in the NTK regime.

EPFL2022

Visually Pleasing Panoramas

Dunai Fuentes Hitos

In this semester project we develop an automated system to rank panoramas captured by the EPFL Livecam according to how visually appealing they are. In other words, a system able to predict an average of the rating of human users for such panoramas. As a way to achieve it, we explore the capabilities of deep neural networks to solve the problem given three different datasets: a general dataset for aesthetics (Photo.net), a custom dataset built with panoramas extracted from Flickr, and a domain specific dataset built from evaluated panoramas extracted from the camera itself. We show that shallow network architectures, limited computational resources, and scarce data (as opposed to highlighted literature on deep nets), are sufficient to provide satisfactory rankings. Finally a roadmap for further development is presented. LCAV1558050917

2016

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Nikolaos Dimitriadis

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphological classifier as a Difference-of-Convex optimization problem and extend this method to multiclass tasks. We then focus on general morphological networks trained with gradient descent variants and show, quantitatively via pruning schemes as well as qualitatively, the sparsity of the resulted representations compared to FeedForward networks with ReLU activations as well as the effect the training optimizer has on such compression techniques. Finally, we show how morphological networks can be employed to guarantee monotonicity and present a softened version of a known architecture, based on Maslov Dequantization, which alleviates issues of gradient propagation associated with its "hard" counterparts and moderately improves performance.

IEEE2021

Theory of Deep Learning: Neural Tangent Kernel and Beyond

Arthur Ulysse Jacot-Guillarmod

EPFL2022