EEG-based Recognition of Covert Attentional Shifts

Attentional shifts precede most of the perceptual processes, and therefore their recognition may constitute an interesting process for both the study of human perception as well as the design of EEG-based Brain Computer Interfaces (BCI). Several studies have focused on EEG modulations of spatial attention, pointing out to changes in both Alpha and Gamma bands (Thut et al. 2006 J. Neurosci 26:9494; Tallon-Baudry and Bertrand 1999 T Cogn Sci 3:151; Jensen et al. 2007 TINS 30:317). This work aims at further characterization of EEG correlates of attention shifts to evaluate their potential use in BCI applications. We recorded 32-channel scalp EEG while subjects covertly shift their attention towards one out of several spatial targets. Consistently with previous studies, characterization of the spatio-temporal patterns using time-frequency analysis (Morlet wavelet transformation), shows main modulations in the alpha and gamma frequency bands. Moreover, it is also shown that these modulations are characterized by intermittent episodes of activity (Palix et al. 2006 Perception 25:234). Furthermore, we assess the EEG-based recognition of spatial attention -time and target location of the shift- using a machine learning approach based on the detection of informative, episodic activity. The proposed approach is based on the hypothesis that recognition may be improved by identifying the moments where the underlying neural phenomena is more salient (e. g. the appearance of episodic oscillations related to attentional shifts), and performing the classification based mainly -if not exclusively- on the activity during these periods. Our results show accurate recognition of attentional shifts (>75%) based on the scalp EEG signal. Moreover, the proposed approach allow us to efficiently detect endogenous processes loosely synchronized to external events. These results provides evidence supporting the feasibility of using attentional processes in Brain-Computer Interfaces.

Analysis of Anticipation Related Potentials for Brain Computer Interaction

Gangadhar Garipelli

Anticipation is a mental process during which a person actively engages in a phase required for the sensory perception and execution of the optimal actions at the arrival of the relevant future events. Since this process occurs before the execution of an intended action, it may be used as a control signal for Brain Computer Interface (BCI) applications. Recognition of neural correlates of this process can enhance the performance of a BCI and in turn reduce mental workload of its users. To this end, it is vital to understand the neural correlates involved in this process and to design robust methods for its recognition in single trials. The analysis of these correlates may also contribute to the basic knowledge of the mechanisms underlying this behavior. The thesis provides three major contributions: (i) it reports methods for the robust recognition of anticipation related Electroencephalogram (EEG) potentials (ii) it provides insights into the selection of appropriate preprocessing steps required for enhancing the Signal-to-Noise Ratio (SNR) of anticipatory slow cortical potentials (SCPs) and (iii) it identifies scalp area specific oscillatory activity related to different aspects of anticipatory behavior. First, we focus on methods for the single trial recognition of anticipatory SCPs using the widely known classical contingent negative variation (CNV) paradigm. Using this paradigm, we demonstrate the feasibility of recognizing the anticipatory SCPs (CNV potentials) using features thatmodel its temporal pattern. We propose a Bayesian approach that exploits temporal evolution and redundancy to quickly classify (e.g., within half of the anticipatory period), without compromising classification accuracy. We then improve upon these recognition rates by using a source localization technique based on the biophysical model of the human head. We further validate the feasibility of recognizing CNV potentials in an online experiment, and report for the first time that, under controlled conditions, these potentials can be reproduced and recognized in realistic interaction scenarios (assistive technology web-browsing) with high accuracies. Second, the thesis provides insights into the selection of appropriate preprocessing stages required for improving the SNR of SCPs. The CNV potentials are characterized by low frequencies that are usually recorded with full-DC, and hence suffer from task-irrelevant high amplitude fluctuations and spatial noise. To account for this, we identified appropriate spectral and spatial filters to improve the SNR. We demonstrate the potential of these preprocessing stages by using fusing multiple electrode specific linear classifiers, which achieve recognition performances of 90±2% (area under curve of receiver operating characteristic), where the classifiers are trained using recordings from one day and tested on the recordings from several days apart. Finally, the thesis identifies different facets of anticipatory behavior. Apart from the widely known CNV potentials, it is not clear which other spectral bands could be related to anticipatory behavior. Using recordings from an experiment (i.e., the assistive technology web browser) where multiple warning stimuli predicted an imperative stimulus, we explored the phase and amplitude response of various oscillatory sub-bands for the identification of markers that could be associated with different aspects of anticipation. From this study we report that: (i) there are duration (4-10 seconds) specific changes in Electroencephalography (EEG) activity in the range 0-1 Hz in the central electrodes correlate with reaction time, (ii) there exist slow oscillations (0.1-1 Hz) in the central electrodes that exhibit phase tuning up to 4 seconds before the onset of a target cue, (iii) there are delta oscillations (1.5-3 Hz), which are entrained to predictive rhythmic warning cues, (iv) there is a selective modulation (increase or decrease) in the amplitude of occipital alpha band (8-12 Hz) based on the relevance of forthcoming visual cue, and (v) there exist a reduction in the beta band (14-30 Hz) amplitude in the sensory motor and association areas lasting up to 10 seconds. The phase tuning and entrainment resulted in a low variance of phase values at the arrival of the imperative stimulus, which may be required for its optimal processing. The amplitude modulation of alpha band activity is likely to be a resultant of sensory suppression and attention. The reduction of beta-band activity over long periods of time suggests holding of sensory-motor association areas until the execution of a planned action. We believe that these observations are the consequence of the endogenous drive on the ongoing oscillations to enhance the processing of the forthcoming stimuli and preparation for an intended action. In summary, the thesis provides methods for the recognition of anticipatory SCPs by exploiting spectral and spatio-temporal characteristics with high performances. By exploring various other oscillatory sub-bands, the spectral characteristics of different aspects of anticipatory behavior are also identified. Further, methods modeling these characteristics can bring forth more robust and faster techniques for BCI systems.

EPFL2011

Emotion Detection and Recognition based on Brain and Peripheral Physiological Signals

Eleni Kroupi

Emotions affect and determine social relationships and interactions, memory and creativity, and influence the mechanisms of rational thinking and decision making. The influence of emotion on decision making has gained attention in computer science. By detecting and recognizing emotions in an automatic way, machines endeavor to ease interaction between users and multimedia content. Automatic emotion detection and recognition can be carried out through analysis of users' various behavioural, and physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), electrodermal activity (EDA), and respiration, among others. These modalities have been extensively studied individually. However, since different individuals may experience the same emotion but express it differently, the various modalities are considered complementary, and fusion of various physiological and behavioural responses is expected to improve the quality of emotion recognition systems. Nevertheless, although representative features and their multimodal integration have been studied in affective computing research for various applications, patterns that arise from the dynamic interrelation among various modalities during emotional processes have received less attention. By summarizing each physiological signal only in a number of features, one may lose information present in the underlying dynamical co-evolution of various physiological signals. Considering all these issues, this thesis aims at detecting and recognizing emotion through brain and peripheral signals, targeting three complementary topics that have not been thoroughly explored. The first one includes emotion assessment from music video clips, the second one emotion assessment from odors, and the third one Quality of Experience (QoE) assessment from two-dimensional (2D) and three-dimensional (3D) video contents, using in all cases brain and peripheral physiological signals. Regarding emotion assessment from music video clips, subject-dependent and subject-independent analyses are carried out in this thesis, and the results reveal that although there are differentiations among the subjects' brain activation patterns, there are still common patterns across them. Moreover, the dynamical co-evolution between EEG and EDA is explored during emotional processes, and the results reveal that the coupling between EDA and EEG of the temporal lobe increases when strong emotions occur with respect to neutral ones. Finally, possible clustering patterns across subject-categories are investigated, and the results reveal that there are common characteristics across subject-categories related to their emotions. Regarding emotion assessment from odors, since the primary response to odors is related to pleasantness perception, which has not yet been thoroughly investigated, this thesis explores the way perceived odor pleasantness influences brain and periphery. In particular, two independent classifiers are trained and tested, one using EEG and the other using ECG features. The results reveal that it is possible to assess odor pleasantness perception from EEG and less accurately from ECG features, in a subject-independent framework. Also, decision fusion of the EEG and ECG classifiers is shown to discriminate odor pleasantness perception. Moreover, in order to explore the dynamical co-evolution between brain and peripheral signals, the coupling between heart rate and EEG is investigated. The results reveal that there is a significant increase in the coupling between ECG and temporal lobe EEG, when pleasant or unpleasant odors are experienced with respect to neutral ones. Enhanced QoE from multimedia contents targets to increase users' sensation of reality, in order to induce stronger emotions and render the user more involved in the experience. In this thesis, QoE is investigated in terms of four aspects, namely perceived depth, perceived overall quality, content preference, and sensation of reality. In particular, it is revealed that it is possible to recognize perceived depth, content preference, and sensation of reality from EEG signals, but not from the peripheral ones. Also, fusion between peripheral and EEG features is found to improve the performance in some cases. Finally, the left frontal cortex seems to be activated when sensation of reality is high, indicating that high sensation of reality is related to approach-related emotional processes. Although the three topics are independently explored in this thesis, their potential integration would endeavor to create immersive multimedia systems, which could adapt their properties according to users' emotions, and enhance, thus, user-experience.

EPFL2014

Towards everyday brain-computer interfacing through decoding visual cognition

Ruslan Aydarkhanov

Decoding visual cognition from non-invasive measurements of brain activity has shown valuable applications. Vision-based Brain-Computer Interfaces (BCI) systems extend from spellers to database search and spatial navigation. Despite the high performance of these systems, they are limited to the laboratory environment. Several attempts were made to improve the quality of life of patients with severe neurodegenerative disorders by helping them acquire more independence. Bringing vision-based BCI to everyday life would benefit a wide range of population from handicapped to healthy people. However, real-life applications impose additional challenges on obtaining neural signals with a good signal-to-noise ratio. The natural environments are rich, dynamic and ambiguous that challenges the visual perception and, therefore, the decoding of underlying neural correlates. The limited attentional resources require to explore and visually sample the environment by freely moving eyes and fixating on the relevant objects. Moreover, everyday activities are rarely isolated leading to a mixture and complex interactions between neural correlates in the measured signal. In this thesis, I explore various aspects of decoding brain activity by developing new protocols for realistic scenarios with challenging visual tasks. The tasks include free visual exploration, dynamic scenes, and multiple tasks, for example, recognition of a billboard category during simulated driving. The major efforts have been directed towards synchronous decoding of Event-Related Potentials (ERP) acquired by the means of electroencephalography (EEG) and Eye-tracking technology, which allowed the extraction of Eye-Fixation Related Potentials (EFRP). Additionally, I investigate the neural correlates of perceptual decision making which is tightly linked to visual recognition. The perceptual challenges can lead to higher temporal variability of obtained ERP which was also addressed from the decoding perspective. The obtained results, firstly, provide new insights on how to tackle the temporal variability of ERP. Secondly, they raise new questions on studying and decoding neural correlates of perceptual decision making. And thirdly, they show the feasibility of decoding visual recognition in a simulated realistic scenario of car driving. Although, as expected, I found a lower performance compared to classical ERP protocols, these findings hold promise and raise new questions to be investigated in order to improve the quality of decoding visual cognition for everyday application.

EPFL2020