Learning with Surprise

Everybody knows what it feels to be surprised. Surprise raises our attention and is crucial for learning. It is a ubiquitous concept whose traces have been found in both neuroscience and machine learning. However, a comprehensive theory has not yet been developed that addresses fundamental problems about surprise: (1) surprise is difficult to quantify. How should we measure the level of surprise when we encounter an unexpected event? What is the link between surprise and startle responses in behavioral biology? (2) the key role of surprise in learning is somewhat unclear. We believe that surprise drives attention and modifies learning; but, how should surprise be incorporated, in general paradigms of learning? and (3) can we develop a biologically plausible theory that explains how surprise can be neurally calculated and implemented in the brain? I propose a theoretical framework to address the above issues about surprise. There are three components to this framework: (1) a subjective confidence-adjusted measure of surprise, that can be used for quantification purposes, (2) a surprise-minimization learning rule that models the role of surprise in learning by balancing the relative contribution of new and old data for inference about the world, and (3) a surprise-modulated Hebbian plasticity rule that can be implemented in both artificial and spiking neural networks. The proposed online rule links surprise to the activity of the neuromodulatory system in the brain, and belongs to the class of neo-Hebbian plasticity rules. My work on the foundations of surprise provides a suitable framework for future studies on learning with surprise. Reinforcement learning methods can be enhanced by incorporating the proposed theory of surprise. The theory could ultimately become interesting for the analysis of fMRI and EEG data. It may also inspire new synaptic plasticity rules that are under the simultaneous control of reward and surprise. Moreover, the proposed theory can be used to make testable predictions about the time course of the neural substrate of surprise (e.g., noradrenaline), and suggests behavioral experiments that can be performed on real animals for studying surprise-related neural activity.