Capsule networks as recurrent models of grouping and segmentation

Classically, visual processing is described as a cascade of local feedforward computations. Feedforward Convolutional Neural Networks (ffCNNs) have shown how powerful such models can be. However, using visual crowding as a well-controlled challenge, we previously showed that no classic model of vision, including ffCNNs, can explain human global shape processing. Here, we show that Capsule Neural Networks (CapsNets), combining ffCNNs with recurrent grouping and segmentation, solve this challenge. We also show that ffCNNs and standard recurrent CNNs do not, suggesting that the grouping and segmentation capabilities of CapsNets are crucial. Furthermore, we provide psychophysical evidence that grouping and segmentation are implemented recurrently in humans, and show that CapsNets reproduce these results well. We discuss why recurrence seems needed to implement grouping and segmentation efficiently. Together, we provide mutually reinforcing psychophysical and computational evidence that a recurrent grouping and segmentation process is essential to understand the visual system and create better models that harness global shape computations.

Crowding and the Architecture of the Visual System

Adrien Christophe Doerig

Classically, vision is seen as a cascade of local, feedforward computations. This framework has been tremendously successful, inspiring a wide range of ground-breaking findings in neuroscience and computer vision. Recently, feedforward Convolutional Neural Networks (ffCNNs), a kind of deep neural network inspired by this classic framework, have revolutionized computer vision and been adopted as tools in neuroscience. However, despite these successes, there is much more to vision. First, there are flagrant architectural differences between biological systems and the classic framework. For example, recurrence is abundant in the brain but absent from the classic framework and ffCNNs. Although there is widespread agreement about the importance of these recurrent connections, their computational role is still poorly understood. Second, these architectural differences lead to behavioural differences too, highlighted by psychophysical evidence. Relatedly, ffCNNs are extremely vulnerable to small changes to their inputs and do not generalize well beyond the dataset used to train them. Human vision, in contrast, is much more robust. New insights are needed to face up to these challenges. In this thesis, I use visual crowding and related psychophysical effects as probes into visual processes that go beyond the classic framework. In crowding, perception of a target deteriorates in clutter. I focus on global aspects of crowding, in which perception of a small target is strongly modulated by the global configuration of elements across the visual field. I show that models based on the classic framework, including ffCNNs, cannot explain these effects for principled reasons and identify recurrent grouping and segmentation as a key missing ingredient. Then, I show that capsule networks, a recent kind of deep learning architecture combining the power of ffCNNs with recurrent grouping and segmentation, naturally explain these effects. I provide psychophysical evidence that humans indeed use a similar recurrent grouping and segmentation strategy in global crowding effects. In crowding, visual elements interfere across space. To study how elements interfere over time, I use the Sequential Metacontrast psychophysical paradigm, in which perception of visual elements depends on elements presented hundreds of milliseconds later. I psychophysically characterize the temporal structure of this interference and propose a simple computational model. My results support the idea that perception is a discrete process. I lay out theoretical implications of these findings. Together, the results presented here provide stepping-stones towards a fuller understanding of the visual system by suggesting architectural changes needed for more human-like neural computations.

EPFL2020

How crowding challenges (feedforward) convolutional neural networks

Alban Bornet, Adrien Christophe Doerig, Gregory Francis, Michael Herzog, Ben Henrik Lönnqvist, Lynn Schmittwilken

Are (feedforward) convolutional neural networks (CNNs) good models for the human visual system? Here, we used visual crowding as a well-controlled psychophysical test to probe CNNs. Visual crowding is a ubiquitous breakdown of object recognition in the human visual system, whereby targets become jumbled and unrecognisable in the presence of flanking objects. Humans exhibit several well-documented effects of crowding, such as invariance to size, where the size of the target and flanker letters may be changed without impacting the strength of crowding. We show that feedforward CNNs are unable to reproduce invariance to size, confusion between target and flanker identities, and importantly uncrowding, where paradoxically increasing the number of flankers improves performance. We investigate this phenomenon using a recurrent, neurally inspired model called LAMINART, which we find can reproduce uncrowding as observed in humans. Furthermore, we show that capsule networks, a recurrent family of CNNs with grouping and segmentation mechanisms, outperform any other models of uncrowding to date, demonstrating the importance of grouping and segmentation in mechanisms in visual information processing in general.

2021

Capsule networks explain complex spatial processing

Adrien Christophe Doerig, Michael Herzog, Lynn Schmittwilken

Classically, visual processing is described as a cascade of local feedforward computations. This view has received striking support from the success of convolutional neural networks (CNNs). However, CNNs only roughly mimic human vision. For example, CNNs do not take the global spatial configuration of visual elements into account and thus fail at simple tasks such as explaining crowding and uncrowding. In crowding, the perception of a target deteriorates in the presence of neighboring elements. Classically, adding flanking elements is thought to always decreases performance. However, adding flankers even far away from the target can improve performance, depending on the global configuration (uncrowding). We showed previously that no classic model of crowding, including CNNs, can explain uncrowding. Here, we show that capsule networks, a type of deep network combining CNNs and object segmentation, explain both crowding and uncrowding. We trained capsule networks to recognize targets and groups of shapes. There were no crowding/uncrowding stimuli in the training set. When we subsequently tested the network on crowding/uncrowding stimuli, both crowding and uncrowding occurred. We show theoretically how crowding and uncrowding naturally emerge from neural dynamics in capsule networks. These powerful recurrent models offer a new framework to understand previously unexplained experimental results.

SAGE PUBLICATIONS LTD2019

Crowding and the Architecture of the Visual System

Adrien Christophe Doerig

EPFL2020