Fonction d'activation | EPFL Graph Search

Dans le domaine des réseaux de neurones artificiels, la fonction d'activation est une fonction mathématique appliquée à un signal en sortie d'un neurone artificiel. Le terme de "fonction d'activation" vient de l'équivalent biologique "potentiel d'activation", seuil de stimulation qui, une fois atteint entraîne une réponse du neurone. La fonction d'activation est souvent une fonction non linéaire. Un exemple de fonction d'activation est la fonction de Heaviside, qui renvoie tout le temps 1 si le signal en entrée est positif, ou 0 s'il est négatif. Caractéristiques des fonctions d'activation Les fonctions d'activation sont utilisées selon leurs caractéristiques :

Non-linéarité : Quand une fonction est non linéaire, un réseau neuronal à 2 couches peut être considéré comme un approximateur de fonction universel. Note: La fonction identité a l'effet inverse, rendant un réseau neuronal multicouches équivalent à un réseau neuronal à une mono-couche.
Partout différentiable :

Local in vivo optical measurements of neurometabolic and neurovascular coupling mechanisms in the rodent cortex

Elena Migacheva

This thesis introduces a novel approach for in vivo separation and quantification of spatio-temporal dynamics of optical coefficients (absorption, scattering), oxygen saturation (SO2) and haemoglobin concentrations within rat brain somatosensory cortex under baseline and neuronal stimulation conditions. The quantification of mentioned parameters became possible due to design of a novel optical setup for localized spatio-temporally-resolved (within ∼ 1 mm3 at 10 Hz) reflectance spectroscopy (STRReS) measurements at small source-detector separations. The possibility to automatically take into account the light propagation path lengths in tissue using Monte Carlo modeling, defines the novelty of this approach and its perspectives for brain research. This kind of corrections is currently limited or inaccurate with existing techniques like optical intrinsic signal imaging. In particular, the main aim of this thesis is to explore and better understand the local spatio-temporal characteristics of neurovascular and neurometabolic coupling mechanisms within in vivo rat cerebral cortex at initial and subsequent stages of neuronal stimulation with using STRReS technique. It has been proven theoretically (with Monte Carlo simulations) and experimentally (from in vivo rat brain cortex), that STRReS measurements are suitable from in vivo somatosensory cortex under neuronal stimulation conditions. As a part of this research, we have validated theoretically and experimentally the biological importance of new optical parameter ”gamma” (parameter of the phase function), which is nessesary to take into account for accurate optical coefficients extraction. On the other side, the STRReS technique was not found to be helpful to determine differential factors corrections for improvement of optical intrinsic signal imaging analysis due to differences in experiment’s geometries. Finally, it has been shown that temporal dynamics of reflectance is strongly influenced by light scattering variations within 500-900 nm range. Specifically, the shape of reflectance dynamics at 600 and 610 nm differs significantly from deoxy-haemoglobin concentration dynamics at initial stage of neural activation. For this reason, it is important always to take into account the variations of both components of reflectance signal: absorption and scattering. Additionally, an initial delay of 1 s was found in SO2 and haemoglobin concentrations dynamics. The interpretation of this delay, however, is strongly dependent on the origin of scattering signal. Our results are well correlated with ”metabolic” hypothesis of underlying neurovascular and neurometabolic coupling mechanisms, assuming that the scattering signal has a non-vascular origin at 500-599 nm. Nevertheless, the ”neurogenic” hypothesis is not completely excluded as far as the scattering signal has a vascular origin at 500-599 nm.

EPFL2012

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Nikolaos Dimitriadis

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphological classifier as a Difference-of-Convex optimization problem and extend this method to multiclass tasks. We then focus on general morphological networks trained with gradient descent variants and show, quantitatively via pruning schemes as well as qualitatively, the sparsity of the resulted representations compared to FeedForward networks with ReLU activations as well as the effect the training optimizer has on such compression techniques. Finally, we show how morphological networks can be employed to guarantee monotonicity and present a softened version of a known architecture, based on Maslov Dequantization, which alleviates issues of gradient propagation associated with its "hard" counterparts and moderately improves performance.

IEEE2021

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generalization properties of NAS under a unifying framework enabling (deep) layer skip connection search and activation function search. To this end, we derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime using a certain search space including mixed activation functions, fully connected, and residual neural networks. We use the minimum eigenvalue to establish generalization error bounds of NAS in the stochastic gradient descent training. Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performing architectures, even in the case without training, leading to a trainfree algorithm based on our theory. Accordingly, our numerical validation shed light on the design of computationally efficient methods for NAS. Our analysis is non-trivial due to the coupling of various architectures and activation functions under the unifying framework and has its own interest in providing the lower bound of the minimum eigenvalue of NTK in deep learning theory.

2022

Local in vivo optical measurements of neurometabolic and neurovascular coupling mechanisms in the rodent cortex

Elena Migacheva

EPFL2012