Hyperbolic discounting | EPFL Graph Search

In economics, hyperbolic discounting is a time-inconsistent model of delay discounting. It is one of the cornerstones of behavioral economics and its brain-basis is actively being studied by neuroeconomics researchers. According to the discounted utility approach, intertemporal choices are no different from other choices, except that some consequences are delayed and hence must be anticipated and discounted (i.e., reweighted to take into account the delay). Given two similar rewards, humans show a preference for one that arrives sooner rather than later. Humans are said to discount the value of the later reward, by a factor that increases with the length of the delay. In the financial world, this process is normally modeled in the form of exponential discounting, a time-consistent model of discounting. Many psychological studies have since demonstrated deviations in instinctive preference from the constant discount rate assumed in exponential discounting. Hyperbolic discounting is

Stress, individual differences, and norepinephrine in reinforcement learning-based prediction of mouse behavior in conditioning and spatial learning

Gediminas Luksys, Maria del Carmen Sandi Perez

Acute stress regulates different aspects of behavioral learning through the action of stress hormones and neuromodulators. Stress effects depend on stressor's type, intensity, timing, and the learning paradigm. In addition, genetic background of animals might be an important factor in determining how stress influences their performance. To study these effects I performed conditioning and spatial learning experiments, exposing 2 different genetic strains of mice (C57BL/6 and DBA/2) to a variety of different stressors such as elevated platform, uncertainty in food delivery or in escape platform location, food deprivation, and low water temperature. I first show that these factors have diverse effects on animal behavior in these tasks, often exhibiting nonlinear inter-factor relationships. Mouse behavior in such tasks can be formalized and studied using reinforcement learning (RL) models. In this framework, the effects of stress and genetic background on learning and memory could be attributed to differential dynamics of model meta-parameters (such as learning rate, balance between exploitation and exploration, and future reward discounting), which are thought to be related to the activity of different brain systems, particularly the neuromodulators. In both conditioning and spatial learning experiments, RL models were shown to be flexible enough to reproduce differences in learning and performance that are present between different experimental groups. In order to consider how the combination of modulatory factors such as stress, genetic strain, motivation, anxiety, and manipulations of norepinephrine levels affect animal behavior, I developed a black-box model, which can be trained to use these modulatory factors for determining the suitable meta-parameters of the RL model. Such RL model can successfully predict individual animal behavior in the hole-box conditioning task, given the information about current modulatory factors and animal's previous task performance. In order to study neurobiological correlates, I also performed pharmacological manipulations of adrenergic alpha-2 receptors. Results from these manipulations provide computational insights into how norepinephrine affects performance accuracy and future reward discounting to produce the inverted-U-shape relation between norepinephrine levels and behavioral performance. Such approach makes it possible to understand how genetic strain, acute stress, motivation, uncertainty, and norepinephrine affect learning intensity, performance accuracy, and future reward discounting.

EPFL2009

Stress, genotype and norepinephrine in the prediction of mouse behavior using reinforcement learning

Wulfram Gerstner, Gediminas Luksys, Maria del Carmen Sandi Perez

Individual behavioral performance during learning is known to be affected by modulatory factors, such as stress and motivation, and by genetic predispositions that influence sensitivity to these factors. Despite numerous studies, no integrative framework is available that could predict how a given animal would perform a certain learning task in a realistic situation. We found that a simple reinforcement learning model can predict mouse behavior in a hole-box conditioning task if model metaparameters are dynamically controlled on the basis of the mouse's genotype and phenotype, stress conditions, recent performance feedback and pharmacological manipulations of adrenergic alpha-2 receptors. We find that stress and motivation affect behavioral performance by altering the exploration-exploitation balance in a genotype-dependent manner. Our results also provide computational insights into how an inverted U-shape relation between stress/arousal/norepinephrine levels and behavioral performance could be explained through changes in task performance accuracy and future reward discounting.

Nature Publishing Group2009

Encoding of Marginal Utility across Time in the Human Brain

Marginal utility theory prescribes the relationship between the objective property of the magnitude of rewards and their subjective value. Despite its pervasive influence, however, there is remarkably little direct empirical evidence for such a theory of value, let alone of its neurobiological basis. We show that human preferences in an intertemporal choice task are best described by a model that integrates marginally diminishing utility with temporal discounting. Using functional magnetic resonance imaging, we show that activity in the dorsal striatum encodes both the marginal utility of rewards, over and above that which can be described by their magnitude alone, and the discounting associated with increasing time. In addition, our data show that dorsal striatum may be involved in integrating subjective valuation systems inherent to time and magnitude, thereby providing an overall metric of value used to guide choice behavior. Furthermore, during choice, we show that anterior cingulate activity correlates with the degree of difficulty associated with dissonance between value and time. Our data support an integrative architecture for decision making, revealing the neural representation of distinct sub-components of value that may contribute to impulsivity and decisiveness.

2009

Stress, individual differences, and norepinephrine in reinforcement learning-based prediction of mouse behavior in conditioning and spatial learning

Gediminas Luksys, Maria del Carmen Sandi Perez

EPFL2009