Reward boosts reinforcement-based motor learning

Besides relying heavily on sensory and reinforcement feedback, motor skill learning may also depend on the level of motivation experienced during training. Yet, how motivation by reward modulates motor learning remains unclear. In 90 healthy subjects, we investigated the net effect of motivation by reward on motor learning while controlling for the sensory and reinforcement feedback received by the participants. Reward improved motor skill learning beyond performance-based reinforcement feedback. Importantly, the beneficial effect of reward involved a specific potentiation of reinforcement-related adjustments in motor commands, which concerned primarily the most relevant motor component for task success and persisted on the following day in the absence of reward. We propose that the long-lasting effects of motivation on motor learning may entail a form of associative learning resulting from the repetitive pairing of the reinforcement feedback and reward during training, a mechanism that may be exploited in future rehabilitation protocols.

Two-photon calcium imaging of neocortical projection neurons in whisker somatosensory cortex during goal-directed sensorimotor learning

Dimitra Angeliki Vavladeli

Abstract Excitatory projection neurons of the neocortex are thought to play important roles in per-ceptual and cognitive functions of the brain by directly connecting diverse cortical and subcortical areas. However, many aspects of the anatomical and functional organization of these inter-areal connections are unknown. The mouse primary somatosensory whisk-er barrel cortex (S1) serves as an important model for investigating the mammalian neo-cortex, and, here, I firstly investigate the structure and secondly the function of a specific subset of S1 cortico-cortical long-range projection neurons. In the first part of my thesis, I studied long-range axonal projections of excitatory layer 2/3 neurons with cell bodies located in S1. As a population, these neurons densely projected to secondary whisker somatosensory cortex (S2) and primary/secondary whisker motor cortex (M1/2), with additional axon in the dysgranular zone surrounding the barrel field, perirhinal temporal association cortex and striatum. The execution of a goal-directed behavior requires the brain to process incoming sensory information from the environment in a context-, learning- and motivation-dependent manner in order to perform specific motor actions. Cortico-cortical communica-tion in the context of goal-directed sensorimotor transformation has begun to be studied, but little is known about how signaling between interconnected cortical areas is modified by sensorimotor learning, as well as in response to changes in reward contingencies. Hence, in the second part of my thesis, I studied cortico-cortical dynamics in primary whisker somatosensory barrel cortex (S1) of mice during a combined whisker and audito-ry task. Using transgenic mice expressing GCaMP6f combined with two-photon micros-copy and retrograde labeling techniques, I chronically monitored the activity of excitatory layer 2/3 neurons in S1 projecting to M1 or S2, while mice learned the behavioral switch task. The results demonstrated that both classes of neurons responded after whisker and auditory stimulation. However, the whisker stimulus evoked response was stronger than the auditory stimulus evoked response. Neurons projecting to S2 exhibited stronger re-sponses compared to neurons projecting to M1 neurons. Those responses remained rela-tively stable across training sessions and under different reward conditions. Furthermore, both classes of neurons responded during spontaneous licking, but neurons projecting to S2 had larger licking-related responses compared to neurons projecting to M1. This work therefore furthers our knowledge of the structure and function of specific types of cortical projection neurons, which is a necessary step towards detailed under-standing of how sensory information might be signaled from primary sensory areas to downstream brain regions for further processing.

EPFL2018

Models of Reward-Modulated Spike-Timing-Dependent Plasticity

Nicolas Frémaux

How do animals learn to repeat behaviors that lead to the obtention of food or other “rewarding” objects? As a biologically plausible paradigm for learning in spiking neural networks, spike-timing dependent plasticity (STDP) has been shown to perform well in unsupervised learning tasks such as receptive field development. However, STDP fails to take behavioral relevance into account, and as such is inadequate to explain a vast range of learning tasks in which the final outcome, conditioned on the prior execution of a series of actions, is signaled to an animal through sparse rewards. In this thesis, I show that the addition of a third, global, reward-based factor to the pre- and postsynaptic factors of STDP is a promising solution to this problem, consistent with experimental findings. One one hand, dopamine is a neuromodulator which has been shown to encode reward signals in the brain. On the other hand, STDP has been shown to be affected by dopamine, even though the precise nature of the interaction is unclear. Moreover, the theoretical framework of reinforcement learning provides strong foundation for the analysis of these learning rules. After studying existing examples of such rules in a navigation task, I derive simple functional requirements for reward-modulated learning rules, and illustrate these in a motor learning task. One of those functional requirements is the existence a “critic” structure, constantly evaluating the potential for rewarding events. The implication of the existence of such a critic on the interpretation of psychophysical experiments are also discussed. Finally, I propose a biologically plausible implementation of such a structure, that performs motor or navigational tasks. This is based on a generalization of temporal difference learning, a well-known reinforcement learning framework, to continuous time, well suited to an implementation with spiking neurons. These result provide a unified picture of reward-modulated learning rules: even though different rules have been proposed, these can be reduced to a single model at the synaptic level, with variations in the computation of the neuromodulatory signal enabling switching between different learning rules.

EPFL2013

Can Daytime Napping Assist the Process of Skills Acquisition After Stroke?

Winifried Backhaus, Friedhelm Christoph Hummel

Acquisition and reacquisition of skills is a main pillar of functional recovery after stroke. Nighttime sleep has a positive influence on motor learning in healthy individuals, whereas the effect of daytime sleep on neuro-rehabilitative training and relearning of the trained skills is often neglected. The aim of this study was to investigate the relationship between daytime sleep (napping) and the ability to learn a new visuomotor task in chronic stroke patients. The main hypothesis was that sleep enhances motor memory consolidation after training resulting in better motor performance after a period of daytime sleep. Thirty stroke survivors completed the study. They were randomized to one of three different conditions (i) wakeful resting, (ii) short nap (10-20 min), or (iii) long nap (50-80 min). All individuals trained the task with the contralesional, stroke-impaired hand, behavioral evaluation was performed after the break time (wake, nap), and 24 h later. Patients demonstrated a significant task-related behavioral improvement throughout the training. In contrast to the main hypothesis, there was no evidence for sleep-dependent motor consolidation early after the initial, diurnal break, or after an additional full night of sleep. In a secondary analysis, the performance changes of stroke survivors were compared with those of a group of healthy older adults who performed the identical task within the same experimental setup with their non-dominant hand. Performance levels were comparable between both cohorts at all time points. Stroke-related difficulties in motor control did not impact on the degree of performance improvement through training and daytime sleep did not impact on the behavioral gains in the two groups. In summary, the current study indicates that one-time daytime sleep after motor training does not influence behavioral gains.