Spatial and temporal integration of visual features

Visual processing can be seen as the integration and segmentation of features. Objects are composed of contours, integrated into shapes and segmented from other contours. Information also needs to be integrated to solve the ill-posed problems of vision. For example, in the "color" perception of an object, illuminance needs to be discounted, requiring large-scale integration of luminance values. Whereas there is little controversy about the crucial role of integration, very little is known about how it really works. In this thesis, I focused on large-scale spatiotemporal information using two paradigms. First, I used the Ternus-Pikler display (TPD) to understand non-retinotopic, temporal integration, and then I used crowding to understand spatial integration across, more or less, the entire visual field. Motions of object parts are perceived relative to the specific object. For example, a reflector on a bicycle wheel seems to rotate even though it is cycloidal in retinotopic coordinates. This is because the reflector's motion is subtracted from the bike's horizontal motion. Instead of bike motion, I used the TPD, which is perfectly suited to understand non-retinotopic processing. There are two possibilities of how information may be integrated non-retinotopically: either based on attentional tracking, e.g., of the reflector's motion, or relying on inbuilt automated mechanisms. I showed that attentional tracking does not play a major role for non-retinotopic processing in the TPD. Second, I showed that invisible retinotopic information can strongly modulate the visible, non-retinotopic percept, further supporting automated integration processes. Crowding occurs when the perception of a target deteriorates because of the surrounding elements. It is the standard situation in everyday vision, since elements are rarely encountered in isolation. The classic model of vision integrates information from low-level to high-level feature detectors. By adding flankers, this model can only predict performance deterioration. However, this prediction was proven wrong because flankers far from the target can even lead to a release of crowding. Integration across the entire visual field is crucial. Here, I systematically investigated the characteristics of this large-scale integration. First, I dissected complex multi-flanker configurations and showed that low-level aspects play only a minor role. Configural aspects and the Gestalt principle of Prägnanz seem to be involved instead. However, as I showed secondly, the basic Gestalt principles fail to explain our results. Lastly, I tested several computational models, including one-stage feedforward models that integrate information within a local area or across the whole visual field, and two-stage recursive models that integrate global information and then explicitly segment elements. I showed that all models fail, unless they take explicit grouping and segmentation processing into accounts, such as capsule networks and the Laminart model.Overall, spatial and temporal integration is rather a complex inbuilt automated mechanism, and integration occurs across the whole visual field, contrary to most classic and recent models in vision. Moreover, global integration can only be reproduced by two-stage models, which process grouping and segmentation. To better understand perception, we need to consider models that group elements by multiple processes and recursively segment other groups explicitly.

The anatomy of object recognition - visual form agnosia caused by medial occipitotemporal stroke

Johannes Rüter

The influential model on visual information processing by Milner and Goodale (1995) has suggested a dissociation between action- and perception-related processing in a dorsal versus ventral stream projection. It was inspired substantially by the observation of a double dissociation of disturbed visual action versus perception in patients with optic ataxia on the one hand and patients with visual form agnosia (VFA) on the other. Unfortunately, almost all cases withVFAreported so far suffered from inhalational intoxication, the majority with carbon monoxide (CO). Since CO induces a diffuse and widespread pattern of neuronal and white matter damage throughout the whole brain, precise conclusions from these patients with VFA on the selective role of ventral stream structures for shape and orientation perception were difficult. Here, we report patient J.S., who demonstrated VFA after a well circumscribed brain lesion due to stroke etiology. Like the famous patient D.F. with VFA after CO intoxication studied by Milner, Goodale, and coworkers (Goodale et al., 1991, 1994; Milner et al., 1991; Servos et al., 1995; Mon-Williams et al., 2001a,b; Wann et al., 2001; Westwood et al., 2002; McIntosh et al., 2004; Schenk and Milner, 2006), J.S. showed an obvious dissociation between disturbed visual perception of shape and orientation information on the one side and preserved visuomotor abilities based on the same information on the other. In both hemispheres, damage primarily affected the fusiform and the lingual gyri as well as the adjacent posterior cingulate gyrus.Weconclude that these medial structures of the ventral occipitotemporal cortex are integral for the normal flow of shape and of contour information into the ventral stream system allowing to recognize objects.

Society for Neuroscience2009

Transcranial Magnetic Stimulation of Early Visual Cortex Reveals a Window of Integration of Substantial Duration.

Michael Herzog, Johannes Rüter, Frank Scharnowski

How the brain achieves integration of temporally dispersed information is one of the enigmas in the neurosciences. By combining feature fusion with transcranial magnetic stimulation (TMS), we show that individual visual features are independently stored for several hundred milliseconds before condensed in a conscious percept. We presented two verniers with opposite offset directions in rapid succession at the same retinotopic location. Because of the short durations (30ms each) only one fused vernier is perceived. The perceived offset of the fused vernier is a combination of the offsets of the two verniers which themselves are consciously not accessible. First, we balanced performance at approximately 50%, so that neither the first nor the second vernier dominated the fused percept. We then applied TMS at different times after the onset of the first vernier over the occipital cortex to interfere with visual processing. When TMS was applied from 45ms to 120ms after the onset of the first vernier, the second vernier dominated. For later onset asynchronies of up to 370ms, the first vernier dominated. Hence, even though the individual verniers are not consciously accessible, they still can be manipulated by TMS for a period of more than 300ms. Our results demonstrate that the individual verniers are stored independently for several hundred milliseconds before they are condensed into a conscious percept. Thus, the brain collects and integrates feature information across a broad time window before consciousness. Acknowledgement: This work was supported by the SNSF (Swiss National Science Foundation).

2007

The geometry of visual perception: Retinotopic and nonretinotopic representations in the human visual system

Michael Herzog

Geometry is closely linked to visual perception; yet, very little is known about the geometry of visual processing beyond early retinotopic organization. We present a variety of perceptual phenomena showing that a retinotopic representation is neither sufficient nor necessary to support form perception. We discuss the popular Bobject files-concept as a candidate for nonretinotopic representations and, based on its shortcomings, suggest future directions for research using local manifold representations. We suggest that these manifolds are created by the emergence of dynamic reference-frames that result from motion segmentation. We also suggest that the metric of these manifolds is based on relative motion vectors.

2010