Effects of stress and genotype on meta-parameter dynamics in reinforcement learning

Stress and genetic background regulate different aspects of behavioral learning through the action of stress hormones and neuromodulators. In reinforcement learning (RL) models, meta-parameters such as learning rate, future reward discount factor, and exploitation-exploration factor, control learning dynamics and performance. They are hypothesized to be related to neuromodulatory levels in the brain. We found that many aspects of animal learning and performance can be described by simple RL models using dynamic control of the meta-parameters. To study the effects of stress and genotype, we carried out 5-hole-box light condition- ing and Morris water maze experiments with C57BL/6 and DBA/2 mouse strains. The animals were exposed to different kinds of stress to evaluate its effects on immediate performance as well as on long-term memory. Then, we used RL models to simulate their behavior. For each experimental session, we estimated a set of model meta-parameters that produced the best ﬁt between the model and the animal performance. The dynamics of several estimated meta-parameters were qualitatively similar for the two simulated experiments, and with statistically signiﬁcant differences between different genetic strains and stress conditions.

Stress, individual differences, and norepinephrine in reinforcement learning-based prediction of mouse behavior in conditioning and spatial learning

Gediminas Luksys, Maria del Carmen Sandi Perez

Acute stress regulates different aspects of behavioral learning through the action of stress hormones and neuromodulators. Stress effects depend on stressor's type, intensity, timing, and the learning paradigm. In addition, genetic background of animals might be an important factor in determining how stress influences their performance. To study these effects I performed conditioning and spatial learning experiments, exposing 2 different genetic strains of mice (C57BL/6 and DBA/2) to a variety of different stressors such as elevated platform, uncertainty in food delivery or in escape platform location, food deprivation, and low water temperature. I first show that these factors have diverse effects on animal behavior in these tasks, often exhibiting nonlinear inter-factor relationships. Mouse behavior in such tasks can be formalized and studied using reinforcement learning (RL) models. In this framework, the effects of stress and genetic background on learning and memory could be attributed to differential dynamics of model meta-parameters (such as learning rate, balance between exploitation and exploration, and future reward discounting), which are thought to be related to the activity of different brain systems, particularly the neuromodulators. In both conditioning and spatial learning experiments, RL models were shown to be flexible enough to reproduce differences in learning and performance that are present between different experimental groups. In order to consider how the combination of modulatory factors such as stress, genetic strain, motivation, anxiety, and manipulations of norepinephrine levels affect animal behavior, I developed a black-box model, which can be trained to use these modulatory factors for determining the suitable meta-parameters of the RL model. Such RL model can successfully predict individual animal behavior in the hole-box conditioning task, given the information about current modulatory factors and animal's previous task performance. In order to study neurobiological correlates, I also performed pharmacological manipulations of adrenergic alpha-2 receptors. Results from these manipulations provide computational insights into how norepinephrine affects performance accuracy and future reward discounting to produce the inverted-U-shape relation between norepinephrine levels and behavioral performance. Such approach makes it possible to understand how genetic strain, acute stress, motivation, uncertainty, and norepinephrine affect learning intensity, performance accuracy, and future reward discounting.

EPFL2009

Elucidating the role of neuron-astrocyte metabolic coupling in learning and memory

Monika Tadi

The brain has very high energy demands that are mainly met by the circulating blood glucose to ensure its proper functioning. Thus, it is not surprising that though the human brain weighs only 2- 3% of the body weight, it consumes approximately 25% of total body glucose. “Neuro-energetic coupling" is a unique feature of the brain and refers to the existence of the tight coupling between (neuronal) synaptic brain activity and energy metabolism. The main excitatory neurotransmitter in the brain is glutamate and most of the energy requirements at the cortical level are met through glutamate-mediated neurotransmission, suggesting that there is a close relationship between brain activity, glutamatergic neurotransmission and energy metabolism. Until the 80s, blood-borne glucose was thought to be the sole energy substrate of the brain cells and it was assumed that glucose is metabolized by neurons and astrocytes (a type of glial cell) independently. It is only during the last two decades that the concept of compartmentalization of energy metabolism between neurons and astrocytes has become more obvious. The transfer of lactate from astrocytes to neurons termed as the “astrocyte-neuron lactate shuttle (ANLS)” model proposed by Dr. Pellerin and Prof. Magistretti posits that glutamate released during brain activation activates glycolysis (one of glucose metabolic pathways) in astrocytes leading to lactate (a monocarboxylate) release in the extracellular space, and this astrocytic lactate is then captured and used as an energy substrate by the activated neurons to meet their energy requirements. It includes a complex chain of events involved in neuro-metabolic coupling and supports the preferential use of lactate by activated neurons under conditions of increased energy demands. Higher brain functions such as learning a new task are associated with structural changes in brain and are metabolically demanding necessitating the need of an additional energy supply to meet the neurons increased energy requirement. In this context, the current thesis project aimed to elucidate the contribution of metabolic coupling between astrocytes and neurons, known as “neuron-astrocyte metabolic coupling” to learning and memory formation. Using an in vivo approach combined with behavioral, molecular and gene manipulation techniques in mice, we set out to investigate the role of neuron-astrocyte metabolic coupling, particularly ANLS in cognition. The project was divided into two main parts: In the first part, the expression of glucose metabolism-related genes was investigated during long term memory (LTM) formation in fear-motivated inhibitory avoidance (IA) learning paradigm. Using (14C) 2-Deoxyglucose (2-DG) technique, we first defined the brain areas that are involved in IA LTM formation. This technique provides an indirect measurement of brain activity by measuring glucose uptake in different brain areas during the task. The results of brain metabolic mapping show an increase in glucose utilization in the hippocampus, amygdala, anterior cingulate cortex and pre-limbic and infra-limbic cortical areas. Results from the quantitative analysis of mRNA levels in the dorsal hippocampus at different time point’s following IA task highlight a time dependent, learning specific change in the expression of genes related to glucose metabolism. Specific set of genes involved in the synthesis and degradation of glycogen (brain glucose reserve present only in astrocytes), pyruvate metabolism and pentose phosphate shunt as well as the ANLS were induced following the IA task. These early observations indicate that the metabolic interactions between astrocytes and neurons undergo modifications during a learning process and suggest that these adaptations may play a key role in the establishment of long-term memory. In the second part of research project, we focused on investigating the behavioral consequences of down regulating the expression of Monocarboxylate Transporter 1 (MCT1), a key ANLS related gene, on cognition and higher brain functions. Mice heterozygous for the MCT1 gene (MCT1+/-) were characterized in numerous behavioral paradigms such as general wellbeing, reflexive and motor capabilities. After having established that MCT1+/- mice have no alterations in general emotional state or in locomotion, a battery of behavioral tests was performed to study cognition in these mice. Compared with wild-type control mice, the MCT1+/- mice display normal muscle strength and motor coordination. However, when tested for higher brain functions, the MCT1+/- mice exhibit learning and memory deficits in several learning paradigms/tasks that are hippocampus and / or amygdala dependent. The cognitive impairment observed in the MCT1 heterozygous mice further highlights the importance of ANLS in memory and suggest that disruption of lactate transfer from astrocytes to neurons via the MCT1 results in cognitive impairment. The overall results in this thesis demonstrate that cerebral energy metabolism, and in particular the transfer of lactate from astrocytes to neurons not only provides temporal adaptations in the learning process, but it also plays a fundamental role in memory formation.

EPFL2013

Surprise-based model estimation in reinforcement learning: algorithms and brain signatures

Vasiliki Liakoni

Learning how to act and adapting to unexpected changes are remarkable capabilities of humans and other animals. In the absence of a direct recipe to follow in life, behaviour is often guided by rewarding and by surprising events. A positive or a negative outcome influences the tendency to repeat some actions, and a sudden unexpected event signals the possible need to act differently or to update one's view about the world. Advances in computational, behavioral, and cognitive neuroscience have indicated that animals employ multiple strategies to learn from interaction. However, our understanding of learning strategies and how they are combined is largely restricted. The main goal of this thesis is to study the use of surprise by ever-adapting biological agents, its contributions to reward-based learning, and its manifestation in the human brain. We first study surprise from a theoretical perspective. In a probabilistic model of changing environments, we show that exact and approximate Bayesian inference give rise to a trade-off between forgetting old observations and integrating them with new ones, modulated by a naturally emerging surprise measure. We develop novel surprised-based algorithms that can adapt in the face of abrupt changes and accurately estimate the model of the world, and that could potentially be implemented in the brain. Next, we focus on the contributions of surprise-based model estimation to reinforcement learning. We couple one of our adaptive algorithms as well as simpler non-adaptive methods with reinforcement learning agents and evaluate their performance on environments exhibiting different characteristics. Abrupt changes that directly affect the agent's policy call for surprise-based adaptation, in order to achieve higher performance. Often, however, the agent does not need to invest in maintaining an accurate model of the environment to obtain high reward levels. More specifically, in stochastic environments or in environments with distal changes, simpler methods, equipped with exploration capacity, perform equally well compared to more elaborate methods. Finally, we turn to human learning behaviour and brain signals of surprise- and reward-based learning. We design a novel sequential decision making task of multiple steps where strategic use of surprising events allows us to dissociate fMRI brain correlates of reward learning and model estimation. We show that Bayesian inference on this task leads to the same surprise measure we found earlier, where the trade-off is now between ignoring new observations and integrating them with the old belief, and we develop reinforcement learning algorithms that perform outlier detection via this surprise-modulated trade-off. At the level of behaviour we find evidence for a model-free policy learning architecture, with potential influences from a model estimation system. At the level of brain responses we identify signatures of both reward- and model estimation signals, supporting the existence of multiple parallel learning systems in the brain. This thesis presents a comparative analysis of surprise-based model estimation methods in theory and simulations, provides insights in the type of approximations that biological agents may adopt, and identifies signatures of model estimation in the human brain. Our results may aid future work aiming at building efficient adaptive agents and at understanding the learning algorithms and the surprise measures implemented in the brain.

EPFL2021