Anticipation- and Error-related EEG Signals during Realistic Human-Machine interaction: A Study on Visual and Tactile Feedback

The exploitation of EEG signatures of cognitive processes can provide valuable information to improve interaction with brain actuated devices. In this work we study these correlates in a realistic situation simulated in a virtual reality environment. We focus on cortical potentials linked to the anticipation of future events (i.e. the contingent negative variation, CNV) and error-related potentials elicited by both visual and tactile feedback. Experiments with 6 subjects show brain activity consistent with previous studies using simpler stimuli, both at the level of ERPs and single trial classification. Moreover, we observe comparable signals irrespective of whether the subject was required to perform motor actions. Altogether, these results support the possibility of using these signals for practical brain machine interaction.

Preventing Breaks in Embodiment in Immersive Virtual Reality

Thibault Serge Mario Porssut

Virtual reality (VR) is immersive not only because of visual integration, but because we can act and perform in a virtual environment (VE). Beyond the mere fact that a VR system provides images computed directly from the users' viewpoint in stereoscopy, it can also provide a natural way to interact within the VE using the entire body. Full-body motion capture allows the user's movement to be mapped to a virtual body. Such a mapping may let users feel in control of (sense of agency), own (sense of body ownership) and locate themselves inside this virtual body (sense of self-location), thus leading them to feel embodied in this virtual body whenever these conditions are met (sense of embodiment (SoE)). Hence the integration of a full body avatar in VR is essential, as it brings even more; experiencing being someone else, train to recover mobility or having superhuman abilities. Most importantly, the avatar mediates the virtual nature of the VE to make it seamless for the user. Thus, this thesis focuses on helping the users execute complex movements in VR by applying a distortion to their movements. However, such a distortion is efficient only if users, when assisted, do not experience a break in embodiment (BiE), i.e., do not abruptly lose their sense of embodiment. Moreover, the way the virtual body is perceived is not consistent between users and some users might accept or reject a virtual body more easily than others. Therefore, this thesis explores how to prevent a BiE while distorting users' movement and adapting the VR experience to each individual. To this end, we designed a system combining brain-computer interface and machine learning to detect when users experience a BiE, compute users' preferences, and adapt the VR application's parameters. We first identified the user`s threshold for perceiving a movement distortion in some critical contexts, such as articular limits. We designed a new distortion to help users execute a complex movement. We introduced a method based on machine learning to find users' preferences when using our new distortion. We also established a link between the brain's error monitoring mechanism and the cognitive process of embodiment. This link exposed a new implicit EEG marker indicating when users experience a BiE. Finally, we designed a system to calibrate the distortion implicitly based on users' preferences with all these elements. This system first consists of detecting a BiE during a continuous movement thanks to state-of-the-art brain-computer interface algorithms. Our machine learning method then computes users' preferences, and the distortion is adapted dynamically. This original approach demonstrates the use of an implicit feedback loop to help users while preventing any BiE. Our framework opens an exciting perspective for personalized and self-adapting embodiment systems.

EPFL2021

Calibration-Free BCI Based Control

Inaki Asier Iturrate Gil

Recent works have explored the use of brain signals to directly control virtual and robotic agents in sequential tasks. So far in such brain-computer interfaces (BCI), an explicit calibration phase was required to build a decoder that translates raw electroencephalography (EEG) signals from the brain of each user into meaningful instructions. This paper proposes a method that removes the calibration phase, and allows a user to control an agent to solve a sequential task. The proposed method assumes a distribution of possible tasks, and infers the interpretation of EEG signals and the task by selecting the hypothesis which best explains the history of interaction. We introduce a measure of uncertainty on the task and on the EEG signal interpretation to act as an exploratory bonus for a planning strategy. This speeds up learning by guiding the system to regions that better disambiguate among task hypotheses. We report experiments where four users use BCI to control an agent on a virtual world to reach a target without any previous calibration process.

2014

Motor and cognitive brain circuits of brain-machine interactions

Silvia Marchesotti

Brain-machine interfaces (BMIs) allow the user to control an external device such as a robotic arm, a cursor, or an avatar in a virtual world through the real-time decoding of brain signals and without the involvement of the musculoskeletal system. Although BMIs hold great promise for providing motor-impaired patients with means of control and interaction with the external world, the neurocognitive mechanisms that are involved in the use of a BMI remain poorly understood. In addition, BMIs allow researchers to investigate and separate the brain processes underlying cognitive functions from those related to motor control and afferent sensory signals that reliably accompany movements. Thus, BMIs are powerful tools for basic and applied neuroscience research. The present work investigates different stages of a BMI-mediated action and targets the subjective, behavioral, and neural signals of BMI control based on electroencephalography (EEG) signals. More specifically, the main study of this thesis has focused on the development of a multimodal imaging platform that allows EEG-based BMI control inside the magnetic resonance imaging (MRI) scanner and during the acquisition of functional MRI (fMRI). This platform has enabled us to exploit both the high temporal resolution of EEG and the high spatial resolution of fMRI to precisely identify the brain mechanisms underlying BMI control and those reflecting the subjective feeling of being in control over a BMI-action (i.e., the sense of agency, SoA). In this study we found an extended cortico-subcortical network involved in operating a motor-imagery BMI. Overall BMI performance was associated with activity in a set of regions including contralateral premotor cortex and the posterior cingulate cortex. Finally, cortical midline regions and the basal ganglia were involved in the subjective sense of controlling the BMI. In a second study, we further investigated whether the ability to control a BMI relates to the ability to perform motor imagery. Despite decades of technical advances, effective BMI-control remains limited to a subset of users. We show that inter-subject variability in BMI proficiency is associated with differences in motor imagery accuracy as captured by subjective and behavioral measurements, pointing to a prominent role of kinesthetic rather than visual imagery. We also identified enhanced lateralized ÎŒ-band oscillations over sensorimotor cortices during motor imagery in high- versus low-aptitude BMI users. Finally, we developed a novel paradigm for joint BMI actions, allowing two users to be jointly engaged in BMI control; our preliminary data provide evidence in support of the hypothesis that joint actions yield BMI-performance improvement even in absence of a physical connection. Our findings also show that during joint BMI-control the SoA over the imagery-mediated actions is significantly enhanced, and is affected by BMI abilities. This work show the potential of applying a multimodal imaging approach to the field of BMI: exploiting our EEG-BMI-fMRI platform we were able to identify, the neural mechanisms involved in motor and cognitive aspects of imagery-mediated BMI. Furthermore, our results enable us to better understand the subjective and behavioral aspects of BMI-actions, and reveal strategies that could potentially reinforce BMI control. Our findings can ultimately be of relevance for the field of neuroprosthetics and make BMI more accessible to a broader range of users.

EPFL2016