Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning

Wulfram Gerstner, Gediminas Luksys, Maria del Carmen Sandi Perez
Nature Publishing Group, 2009
Journal paper

Abstract

Individual behavioral performance during learning is known to be affected by modulatory factors, such as stress and motivation, and by genetic predispositions that influence sensitivity to these factors. Despite numerous studies, no integrative framework is available that could predict how a given animal would perform a certain learning task in a realistic situation. We found that a simple reinforcement learning model can predict mouse behavior in a hole-box conditioning task if model metaparameters are dynamically controlled on the basis of the mouse's genotype and phenotype, stress conditions, recent performance feedback and pharmacological manipulations of adrenergic alpha-2 receptors. We find that stress and motivation affect behavioral performance by altering the exploration-exploitation balance in a genotype-dependent manner. Our results also provide computational insights into how an inverted U-shape relation between stress/arousal/norepinephrine levels and behavioral performance could be explained through changes in task performance accuracy and future reward discounting.

Official source

https://infoscience.epfl.ch/record/173847?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

Stress, individual differences, and norepinephrine in reinforcement learning-based prediction of mouse behavior in conditioning and spatial learning

Gediminas Luksys, Maria del Carmen Sandi Perez

Acute stress regulates different aspects of behavioral learning through the action of stress hormones and neuromodulators. Stress effects depend on stressor's type, intensity, timing, and the learning paradigm. In addition, genetic background of animals might be an important factor in determining how stress influences their performance. To study these effects I performed conditioning and spatial learning experiments, exposing 2 different genetic strains of mice (C57BL/6 and DBA/2) to a variety of different stressors such as elevated platform, uncertainty in food delivery or in escape platform location, food deprivation, and low water temperature. I first show that these factors have diverse effects on animal behavior in these tasks, often exhibiting nonlinear inter-factor relationships. Mouse behavior in such tasks can be formalized and studied using reinforcement learning (RL) models. In this framework, the effects of stress and genetic background on learning and memory could be attributed to differential dynamics of model meta-parameters (such as learning rate, balance between exploitation and exploration, and future reward discounting), which are thought to be related to the activity of different brain systems, particularly the neuromodulators. In both conditioning and spatial learning experiments, RL models were shown to be flexible enough to reproduce differences in learning and performance that are present between different experimental groups. In order to consider how the combination of modulatory factors such as stress, genetic strain, motivation, anxiety, and manipulations of norepinephrine levels affect animal behavior, I developed a black-box model, which can be trained to use these modulatory factors for determining the suitable meta-parameters of the RL model. Such RL model can successfully predict individual animal behavior in the hole-box conditioning task, given the information about current modulatory factors and animal's previous task performance. In order to study neurobiological correlates, I also performed pharmacological manipulations of adrenergic alpha-2 receptors. Results from these manipulations provide computational insights into how norepinephrine affects performance accuracy and future reward discounting to produce the inverted-U-shape relation between norepinephrine levels and behavioral performance. Such approach makes it possible to understand how genetic strain, acute stress, motivation, uncertainty, and norepinephrine affect learning intensity, performance accuracy, and future reward discounting.

EPFL2009

Our brain continuously self-organizes to construct and maintain an internal representation of the world based on the information arriving through sensory stimuli. Remarkably, cortical areas related to different sensory modalities appear to share the same functional unit, the neuron, and develop through the same learning mechanism, synaptic plasticity. It motivates the conjecture of a unifying theory to explain cortical representational learning across sensory modalities. In this thesis we present theories and computational models of learning and optimization in neural networks, postulating functional properties of synaptic plasticity that support the apparent universal learning capacity of cortical networks. In the past decades, a variety of theories and models have been proposed to describe receptive field formation in sensory areas. They include normative models such as sparse coding, and bottom-up models such as spike-timing dependent plasticity. We bring together candidate explanations by demonstrating that in fact a single principle is sufficient to explain receptive field development. First, we show that many representative models of sensory development are in fact implementing variations of a common principle: nonlinear Hebbian learning. Second, we reveal that nonlinear Hebbian learning is sufficient for receptive field formation through sensory inputs. A surprising result is that our findings are independent of specific details, and allow for robust predictions of the learned receptive fields. Thus nonlinear Hebbian learning and natural statistics can account for many aspects of receptive field formation across models and sensory modalities. The Hebbian learning theory substantiates that synaptic plasticity can be interpreted as an optimization procedure, implementing stochastic gradient descent. In stochastic gradient descent inputs arrive sequentially, as in sensory streams. However, individual data samples have very little information about the correct learning signal, and it becomes a fundamental problem to know how many samples are required for reliable synaptic changes. Through estimation theory, we develop a novel adaptive learning rate model, that adapts the magnitude of synaptic changes based on the statistics of the learning signal, enabling an optimal use of data samples. Our model has a simple implementation and demonstrates improved learning speed, making this a promising candidate for large artificial neural network applications. The model also makes predictions on how cortical plasticity may modulate synaptic plasticity for optimal learning. The optimal sampling size for reliable learning allows us to estimate optimal learning times for a given model. We apply this theory to derive analytical bounds on times for the optimization of synaptic connections. First, we show this optimization problem to have exponentially many saddle-nodes, which lead to small gradients and slow learning. Second, we show that the number of input synapses to a neuron modulates the magnitude of the initial gradient, determining the duration of learning. Our final result reveals that the learning duration increases supra-linearly with the number of synapses, suggesting an effective limit on synaptic connections and receptive field sizes in developing neural networks.

EPFL2016

Abstract Excitatory projection neurons of the neocortex are thought to play important roles in per-ceptual and cognitive functions of the brain by directly connecting diverse cortical and subcortical areas. However, many aspects of the anatomical and functional organization of these inter-areal connections are unknown. The mouse primary somatosensory whisk-er barrel cortex (S1) serves as an important model for investigating the mammalian neo-cortex, and, here, I firstly investigate the structure and secondly the function of a specific subset of S1 cortico-cortical long-range projection neurons. In the first part of my thesis, I studied long-range axonal projections of excitatory layer 2/3 neurons with cell bodies located in S1. As a population, these neurons densely projected to secondary whisker somatosensory cortex (S2) and primary/secondary whisker motor cortex (M1/2), with additional axon in the dysgranular zone surrounding the barrel field, perirhinal temporal association cortex and striatum. The execution of a goal-directed behavior requires the brain to process incoming sensory information from the environment in a context-, learning- and motivation-dependent manner in order to perform specific motor actions. Cortico-cortical communica-tion in the context of goal-directed sensorimotor transformation has begun to be studied, but little is known about how signaling between interconnected cortical areas is modified by sensorimotor learning, as well as in response to changes in reward contingencies. Hence, in the second part of my thesis, I studied cortico-cortical dynamics in primary whisker somatosensory barrel cortex (S1) of mice during a combined whisker and audito-ry task. Using transgenic mice expressing GCaMP6f combined with two-photon micros-copy and retrograde labeling techniques, I chronically monitored the activity of excitatory layer 2/3 neurons in S1 projecting to M1 or S2, while mice learned the behavioral switch task. The results demonstrated that both classes of neurons responded after whisker and auditory stimulation. However, the whisker stimulus evoked response was stronger than the auditory stimulus evoked response. Neurons projecting to S2 exhibited stronger re-sponses compared to neurons projecting to M1 neurons. Those responses remained rela-tively stable across training sessions and under different reward conditions. Furthermore, both classes of neurons responded during spontaneous licking, but neurons projecting to S2 had larger licking-related responses compared to neurons projecting to M1. This work therefore furthers our knowledge of the structure and function of specific types of cortical projection neurons, which is a necessary step towards detailed under-standing of how sensory information might be signaled from primary sensory areas to downstream brain regions for further processing.

EPFL2018

Stress, individual differences, and norepinephrine in reinforcement learning-based prediction of mouse behavior in conditioning and spatial learning

Gediminas Luksys, Maria del Carmen Sandi Perez

EPFL2009

EPFL2016

EPFL2018