Prospective Coding by Spiking Neurons

Animals learn to make predictions, such as associating the sound of a bell with upcoming feeding or predicting a movement that a motor command is eliciting. How predictions are realized on the neuronal level and what plasticity rule underlies their learning is not well understood. Here we propose a biologically plausible synaptic plasticity rule to learn predictions on a single neuron level on a timescale of seconds. The learning rule allows a spiking two-compartment neuron to match its current firing rate to its own expected future discounted firing rate. For instance, if an originally neutral event is repeatedly followed by an event that elevates the firing rate of a neuron, the originally neutral event will eventually also elevate the neuron's firing rate. The plasticity rule is a form of spike timing dependent plasticity in which a presynaptic spike followed by a postsynaptic spike leads to potentiation. Even if the plasticity window has a width of 20 milliseconds, associations on the time scale of seconds can be learned. We illustrate prospective coding with three examples: learning to predict a time varying input, learning to predict the next stimulus in a delayed paired-associate task and learning with a recurrent network to reproduce a temporally compressed version of a sequence. We discuss the potential role of the learning mechanism in classical trace conditioning. In the special case that the signal to be predicted encodes reward, the neuron learns to predict the discounted future reward and learning is closely related to the temporal difference learning algorithm TD(lambda).

Synaptic plasticity across different time scales and its functional implications

Claudia Clopath

Humans and animals learn by modifying the synaptic strength between neurons, a phenomenon known as synaptic plasticity. These changes can be induced by rather short stimuli (lasting, for instance, only a few seconds), yet, in order to be useful for long-term memory, they should remain stable for months or years. Experimentalists study synaptic plasticity by applying a vast variety of protocols. In the present thesis we focus on protocols that fall under two main categories: (i) Those that induce synaptic modifications that last for only a few hours ("early phase" of plasticity) (ii) Those that allow synapses to undergo a sequence of steps that transforms the rapid changes occurring during the "early phase" into a stable memory trace ("late phase" of plasticity). The goal of this thesis is to better understand synaptic plasticity across these different phases, early and late, by creating compact mathematic models to describe the plasticity mechanisms. Our approach allows for a synthetic view of the field as well as the exploration of functional consequences of learning. In this direction, we propose a model for the induction of synaptic plasticity that depends on the presynaptic spike time and nonlinearly on the postsynaptic voltage. The model is able to reproduce a broad range of experimental protocols such as voltage-clamp experiments and spike-timing experiments. Since the voltage is a key element in the model, we describe the neuronal activity by using a compact neuron model that faithfully reproduces the voltage time course of pyramidal neurons. In addition, this model of the induction of synaptic plasticity is combined with a trigger process for protein synthesis, and the final stabilization mechanism in order to describe the "late phase". In this combinatory form, it is able to explain experimental phenomena known as tagging experiments and to make testable predictions. A study of functional consequences of the induction model reveals selectivity in the inputs, independent component analysis computation and a tight relation between connectivity and coding. In parallel a top-down approach finding independent components is used to derive a rate-based learning rule which shows structural correlations with the induction model. This unified model across different time scales allowing the stabilization of synapses is crucial to understand learning and memory processes in animals and humans, and a necessary ingredient for any large-scale model of the brain.

EPFL2009

Theory of representation learning in cortical neural networks

Carlos Stein Naves de Brito

Our brain continuously self-organizes to construct and maintain an internal representation of the world based on the information arriving through sensory stimuli. Remarkably, cortical areas related to different sensory modalities appear to share the same functional unit, the neuron, and develop through the same learning mechanism, synaptic plasticity. It motivates the conjecture of a unifying theory to explain cortical representational learning across sensory modalities. In this thesis we present theories and computational models of learning and optimization in neural networks, postulating functional properties of synaptic plasticity that support the apparent universal learning capacity of cortical networks. In the past decades, a variety of theories and models have been proposed to describe receptive field formation in sensory areas. They include normative models such as sparse coding, and bottom-up models such as spike-timing dependent plasticity. We bring together candidate explanations by demonstrating that in fact a single principle is sufficient to explain receptive field development. First, we show that many representative models of sensory development are in fact implementing variations of a common principle: nonlinear Hebbian learning. Second, we reveal that nonlinear Hebbian learning is sufficient for receptive field formation through sensory inputs. A surprising result is that our findings are independent of specific details, and allow for robust predictions of the learned receptive fields. Thus nonlinear Hebbian learning and natural statistics can account for many aspects of receptive field formation across models and sensory modalities. The Hebbian learning theory substantiates that synaptic plasticity can be interpreted as an optimization procedure, implementing stochastic gradient descent. In stochastic gradient descent inputs arrive sequentially, as in sensory streams. However, individual data samples have very little information about the correct learning signal, and it becomes a fundamental problem to know how many samples are required for reliable synaptic changes. Through estimation theory, we develop a novel adaptive learning rate model, that adapts the magnitude of synaptic changes based on the statistics of the learning signal, enabling an optimal use of data samples. Our model has a simple implementation and demonstrates improved learning speed, making this a promising candidate for large artificial neural network applications. The model also makes predictions on how cortical plasticity may modulate synaptic plasticity for optimal learning. The optimal sampling size for reliable learning allows us to estimate optimal learning times for a given model. We apply this theory to derive analytical bounds on times for the optimization of synaptic connections. First, we show this optimization problem to have exponentially many saddle-nodes, which lead to small gradients and slow learning. Second, we show that the number of input synapses to a neuron modulates the magnitude of the initial gradient, determining the duration of learning. Our final result reveals that the learning duration increases supra-linearly with the number of synapses, suggesting an effective limit on synaptic connections and receptive field sizes in developing neural networks.

EPFL2016

Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail

Nicolas Frémaux, Wulfram Gerstner, Walter Senn, Eleni Vasilaki

Changes of synaptic connections between neurons are thought to be the physiological basis of learning. These changes can be gated by neuromodulators that encode the presence of reward. We study a family of reward-modulated synaptic learning rules for spiking neurons on a learning task in continuous space inspired by the Morris Water maze. The synaptic update rule modifies the release probability of synaptic transmission and depends on the timing of presynaptic spike arrival, postsynaptic action potentials, as well as the membrane potential of the postsynaptic neuron. The family of learning rules includes an optimal rule derived from policy gradient methods as well as reward modulated Hebbian learning. The synaptic update rule is implemented in a population of spiking neurons using a network architecture that combines feedforward input with lateral connections. Actions are represented by a population of hypothetical action cells with strong mexican-hat connectivity and are read out at theta frequency. We show that in this architecture, a standard policy gradient rule fails to solve the Morris watermaze task, whereas a variant with a Hebbian bias can learn the task within 20 trials, consistent with experiments. This result does not depend on implementation details such as the size of the neuronal populations. Our theoretical approach shows how learning new behaviors can be linked to reward-modulated plasticity at the level of single synapses and makes predictions about the voltage and spike-timing dependence of synaptic plasticity and the influence of neuromodulators such as dopamine. It is an important step towards connecting formal theories of reinforcement learning with neuronal and synaptic properties