Reward system | EPFL Graph Search

The reward system (the mesocorticolimbic circuit) is a group of neural structures responsible for incentive salience (i.e., "wanting"; desire or craving for a reward and motivation), associative learning (primarily positive reinforcement and classical conditioning), and positively-valenced emotions, particularly ones involving pleasure as a core component (e.g., joy, euphoria and ecstasy). Reward is the attractive and motivational property of a stimulus that induces appetitive behavior, also known as approach behavior, and consummatory behavior. A rewarding stimulus has been described as "any stimulus, object, event, activity, or situation that has the potential to make us approach and consume it is by definition a reward". In operant conditioning, rewarding stimuli function as positive reinforcers; however, the converse statement also holds true: positive reinforcers are rewarding.

Neural Correlates of Reinforcement Learning: eligibility trace, reward prediction error, novelty and surprise

He Xu

In reinforcement learning (RL), an agent makes sequential decisions to maximise the reward it can obtain from an environment. During learning, the actual and expected outcomes are compared to tell whether a decision was good or bad. The difference between the actual outcome and expected outcome is the prediction error. The prediction error can be categorised into two types: the reward prediction error (RPE) and the state prediction error (SPE), which can serve as teaching signals in reinforcement learning models. Electroencephalogram (EEG) studies have also shown that the RPE can be reflected by a EEG waveform, called the feedback-related negativity (FRN), occurring in the frontal-central brain region, between 250 and 400ms after a reward signal is shown. Most FRN studies use one-step decision-making tasks to study the relationship between FRN amplitude and the RPE. However, everyday reinforcement learning situations come usually with many non-rewarded states and actions until a reward is obtained. The first part of this thesis uses a truly sequential decision making paradigm and aims to answer the question whether the FRN still reflects the RPE in multi-step complex tasks. The state prediction error (SPE) measures how much the agent's expectation on state transitions differs before and after an observation. Novelty and surprise are two types of SPE signals that drive learning when the external reward is not yet provided. However, how novelty and surprise interact and contribute in learning remained un-addressed. In this thesis, I proposed a model combining both novelty and surprise to explain human learning when reward is delayed and sparse. I used a 2-block experimental design to distinguish the effect of novelty and surprise in learning, and studied the neural correlates of novelty and surprise in a sequential decision-making task. I implemented different sequential decision-making tasks to study four RL signals, which are the eligibility trace, the RPE, novelty and surprise. I showed the evidence of eligibility trace in human learning using pupil dilation measurement. With EEG recording, I confirmed that the RPE is reflected in the amplitude of FRN (time window of 280-390ms after the state onset), for both directly rewarded and non-directly rewarded states. I proposed a new RL model, called SurNoR, using novelty as the intrinsic reward and surprise as the learning modulator, to explain human learning where no external reward is provided. The novelty signal is found to be reflected between 80-130ms after the state onset in EEG waveform. The surprise signal occurs later than the novelty signal, which is reflected between 150-210ms after the state onset. By using the sequential decision-making paradigm, this thesis extends the EEG observations of RPE and SPE signals from simple one-step tasks to complex multi-step decision-making tasks.

EPFL2020

Two-photon calcium imaging of neocortical projection neurons in whisker somatosensory cortex during goal-directed sensorimotor learning

Dimitra Angeliki Vavladeli

Abstract Excitatory projection neurons of the neocortex are thought to play important roles in per-ceptual and cognitive functions of the brain by directly connecting diverse cortical and subcortical areas. However, many aspects of the anatomical and functional organization of these inter-areal connections are unknown. The mouse primary somatosensory whisk-er barrel cortex (S1) serves as an important model for investigating the mammalian neo-cortex, and, here, I firstly investigate the structure and secondly the function of a specific subset of S1 cortico-cortical long-range projection neurons. In the first part of my thesis, I studied long-range axonal projections of excitatory layer 2/3 neurons with cell bodies located in S1. As a population, these neurons densely projected to secondary whisker somatosensory cortex (S2) and primary/secondary whisker motor cortex (M1/2), with additional axon in the dysgranular zone surrounding the barrel field, perirhinal temporal association cortex and striatum. The execution of a goal-directed behavior requires the brain to process incoming sensory information from the environment in a context-, learning- and motivation-dependent manner in order to perform specific motor actions. Cortico-cortical communica-tion in the context of goal-directed sensorimotor transformation has begun to be studied, but little is known about how signaling between interconnected cortical areas is modified by sensorimotor learning, as well as in response to changes in reward contingencies. Hence, in the second part of my thesis, I studied cortico-cortical dynamics in primary whisker somatosensory barrel cortex (S1) of mice during a combined whisker and audito-ry task. Using transgenic mice expressing GCaMP6f combined with two-photon micros-copy and retrograde labeling techniques, I chronically monitored the activity of excitatory layer 2/3 neurons in S1 projecting to M1 or S2, while mice learned the behavioral switch task. The results demonstrated that both classes of neurons responded after whisker and auditory stimulation. However, the whisker stimulus evoked response was stronger than the auditory stimulus evoked response. Neurons projecting to S2 exhibited stronger re-sponses compared to neurons projecting to M1 neurons. Those responses remained rela-tively stable across training sessions and under different reward conditions. Furthermore, both classes of neurons responded during spontaneous licking, but neurons projecting to S2 had larger licking-related responses compared to neurons projecting to M1. This work therefore furthers our knowledge of the structure and function of specific types of cortical projection neurons, which is a necessary step towards detailed under-standing of how sensory information might be signaled from primary sensory areas to downstream brain regions for further processing.

EPFL2018

Reward timing matters in motor learning

Julie Duqué, Pierre Theopistos Vassiliadis

Reward timing, that is, the delay after which reward is delivered following an action is known to strongly influence reinforcement learning. Here, we asked if reward timing could also modulate how people learn and consolidate new motor skills. In 60 healthy participants, we found that delaying reward delivery by a few seconds influenced motor learning. Indeed, training with a short reward delay (1 s) induced continuous improvements in performance, whereas a long reward delay (6 s) led to initially high learning rates that were followed by an early plateau in the learning curve and a lower performance at the end of training. Participants who learned the skill with a long reward delay also exhibited reduced overnight memory consolidation. Overall, our data show that reward timing affects the dynamics and consolidation of motor learning, a finding that could be exploited in future rehabilitation programs.

CELL PRESS2022