Multisensory integration | EPFL Graph Search

Multisensory integration, also known as multimodal integration, is the study of how information from the different sensory modalities (such as sight, sound, touch, smell, self-motion, and taste) may be integrated by the nervous system. A coherent representation of objects combining modalities enables animals to have meaningful perceptual experiences. Indeed, multisensory integration is central to adaptive behavior because it allows animals to perceive a world of coherent perceptual entities. Multisensory integration also deals with how different sensory modalities interact with one another and alter each other's processing. General introduction Multimodal perception is how animals form coherent, valid, and robust perception by processing sensory stimuli from various modalities. Surrounded by multiple objects and receiving multiple sensory stimulations, the brain is faced with the decision of how to categorize the stimuli resulting from different objects or events in the physi

Flight of mind: Sensorimotor and multisensory embodiment with aviation robotics, flight simulator, and virtual reality

Myeong Seop Song

Normally, humans experience an 'I' as residing in one's body and as the agent of one's actions. In other words, the self is experienced as being inside the body (self-location), as having a body (self-identification), and as being able to control the person's body (agency). Over the last decade, research in cognitive neuroscience has revealed that vestibular signals contribute to such aspects of the embodied self. On the other hand, the influences of embodiment on vestibular sensations have received scant attention. In my thesis, I highlight how changes in multisensory and sensorimotor embodiment impact vestibular sensations. For this purpose, I merged technologies from photorealistic virtual reality (VR), aviation robotics such as a drone, flight simulator, haptics with knowledge of cognitive neuroscience about embodiment and bodily self-consciousness (BSC). The multidisciplinary approaches provide new paradigms to modulate embodiment and investigate related vestibular sensations based on multisensory and sensorimotor mechanisms of BSC. Also, the approaches reveal the relationship between embodiment and vestibular sensations. In Study 1, we have objectively shown the possibility of the duplication of the self by measuring peripersonal space (PPS). To this aim, we provided participants with real-time feedback from a drone-mounted camera that autonomously followed them while they responded to multisensory stimuli on their body and in their environment and walked forward. Our results demonstrate that this setup induces two spatially distinct PPSs in healthy participants: reaction times for visuotactile stimuli were faster around both the physical body and the observed body. In Studies 2 & 3, we have added a photorealistic avatar and a haptic vest to a robotic flight simulator to enable and investigate personalized immersive flight experiences in healthy humans. Study 2 showed that modulations of embodiment using sensorimotor stimulation resulted in enhanced subjective flying experience, better piloting performance, and learning. In Study 3, we showed that multisensory - visual and tactile - stimulation could further increase embodiment and improve piloting performance. Furthermore, in both studies, we found a positive relationship between embodiment and flight sensations. In Studies 4 & 5, we developed a new multisensory body scanning technology platform to induce and investigate one of the most common experiences in which the center of awareness is dissociated from the physical body: out-of-body experiences (OBEs). OBEs represent an interesting link between embodiment and vestibular sensations as they are characterized by embodiment of an elevated aerial position and perspective in space that is commonly associated with prominent sensations of floating, elevation, and flight. In this study, we have investigated how several sensorimotor and audio-visual stimulation conditions affect subjective sensations of disembodiment, floating, lightness as well as behavioral measures of self-location. In conclusion, my work developed new paradigms to investigate embodiment, BSC, and vestibular processing. My work revealed the influence of embodiment on vestibular sensations and applied these insights between embodiment and vestibular sensations to the field of field engineering by combining state-of-the-art technologies with knowledge of BSC with broad implications for robotics, neuroscience, of immersive VR, as well as experiential

EPFL2020

Multisensory simultaneity judgment and proximity to the body

Marta Lukowska, Jean-Paul Noel, Andrea Serino

The integration of information across different sensory modalities is known to be dependent upon the statistical characteristics of the stimuli to be combined. For example, the spatial and temporal proximity of stimuli are important determinants with stimuli that are close in space and time being more likely to be bound. These multisensory interactions occur not only for singular points in space/time, but over "windows'' of space and time that likely relate to the ecological statistics of real-world stimuli. Relatedly, human psychophysical work has demonstrated that individuals are highly prone to judge multisensory stimuli as co-occurring over a wide range of time-a so-called simultaneity window (SW). Similarly, there exists a spatial representation of peripersonal space (PPS) surrounding the body in which stimuli related to the body and to external events occurring near the body are highly likely to be jointly processed. In the current study, we sought to examine the interaction between these temporal and spatial dimensions of multisensory representation by measuring the SW for audiovisual stimuli through proximal-distal space (i.e., PPS and extrapersonal space). Results demonstrate that the audiovisual SWs within PPS are larger than outside PPS. In addition, we suggest that this effect is likely due to an automatic and additional computation of these multisensory events in a body-centered reference frame. We discuss the current findings in terms of the spatiotemporal constraints of multisensory interactions and the implication of distinct reference frames on this process.

Association for Research in Vision and Ophthalmology (ARVO)2016

Visuo-vestibular mechanisms of bodily self-consciousness

Joachim Jean-Marie Forget

Bodily self-consciousness is linked to multisensory integration and is particularly dependent on vestibular perception providing the brain with the main sensory cues about body motion and location in space. Vestibular and visual inputs are permanently balanced and integrated to encode the most optimal representation of the external world and of the observer in the central nervous system. Vection, an illusory self-motion experience induced only by visual stimuli, illustrates the fact that the visual and the vestibular systems share common neural underpinnings and a similar phenomenology. Optokinetic stimulation inducing vection and direct vestibular stimulation induce whole-body motion sensations that can be used to explore multisensory interactions. A failure in visuo-vestibular integration, artificially induced by the methods of cognitive psychology or in pathological conditions, has also been reported to altered own body perception and bodily self-consciousness. The respective contributions of the vestibular and visual systems to bodily self-consciousness amongst other polymodal sensory mechanisms, and the neural correlates of visuo-vestibular convergence, should be better understood. We first performed a neuroimaging study of brain regions where optokinetic and vestibular stimuli converge, using 7T functional magnetic resonance imaging in individual subjects. We identified three main regions of convergence: (1) the depth of supramarginal gyrus or retroinsular cortex, (2) the surface of supramarginal gyrus at the temporo-parietal junction, (3) and the posterior part of middle temporal gyrus and superior temporal sulcus. Then, we aimed to induce the embodiment of an external fake rubber hand through visuo-tactile conflict - the so-called rubber hand illusion paradigm, and studied how this integration is modulated by vection. Subjects experiencing vection in the direction of the rubber hand mislocalised the position of their real hand towards the rubber hand indicating that visuo-vestibular stimuli can enhance visuo-tactile integration. We also investigated if visuo-proprioceptive and tactile integration in peripersonal space could be dynamically updated based on the congruency of visual and proprioceptive feedback. A pair of rubber hands or feet provided visual feedback. Fake and real limbs were crossed or uncrossed. We showed that sensory cues were integrated in peripersonal space, dynamically reshaped but only for hands. Finally, we investigated a rare case of an illusory own body perception in an epileptic patient suffering from multiple daily disembodiments during seizures. Seizures were associated to a focal cortical microdysplasia juxtaposed to a developmental venous anomaly in the left angular gyrus, a brain region known to be important for visuo-vestibular integration and bodily self-consciousness. Our results characterize the inferior parietal lobule as a crucial structure in merging visual, vestibular, tactile and proprioceptive inputs, allowing the emergence of the global and unified experience of being âI.â Multisensory body representation can be reshaped transiently using visual and vestibular signals or in relation to a medical condition affecting the temporo-parietal junction. The integration of visual and vestibular signals, aims to adapt dynamically our internal representations to constant changes occurring in our environment.

EPFL2015