Brain-Computer Interface in Multimedia Systems

A brain-computer interface (BCI) is a system that allows a user to communicate with the environment only through cerebral activity, without using muscular output channels. To establish a direct link between the brain and a computer, the electroencephalogram (EEG) signal of a user is measured and then analyzed with the help of signal processing and machine learning algorithms. Once a certain mental activity has been detected by the computer, a response can be displayed on a screen or a command can be sent to a peripheral device, for example a wheelchair or a television. The main application area for BCI is assistive technology for handicapped people. For example, one can imagine artificial limbs controlled by a BCI, a BCI-based spelling device, or an environment control system based on a BCI. Nevertheless, during the last years it has been convincingly shown that communication via a BCI is feasible for able-bodied as well as handicapped users and several new applications such as entertainment and gaming, and neuro-feedback have been proposed for BCI systems. This thesis describes the author’s contributions to the research on brain computer interface systems, focusing on exploring and studying new applications of BCIs especially in multimedia communication domain. After providing a general overview of the EEG signal processing techniques and current BCI systems, three topics of interest have been identified. These topics include BCI for salient image retrieval and image triage, BCI for affect recognition during multimedia consumption, and BCI for studying the EEG correlates of pleasant and unpleasant odors. In the first topic, we present a BCI system, capable of identifying interesting images in an image database. Using this system, images of a database are presented to several users at a fast rate, and the information on whether a given image is salient, is implicitly extracted by means of processing the EEG signals, acquired when users are watching the image sequence. Furthermore, in order to investigate the impact of expertise on a BCI-based salient image retrieval system, the changes in the EEG signals of expert and novice users (with respect to image content) are studied. We show that it is possible to define experimental protocol and judiciously apply signal processing tools such that a practical BCI system can be set up for detection of interest in users during the watching of image sequences. Furthermore, we show that a relatively high retrieval accuracy can be obtained for most of the users of this system. The second presented topic deals with an affective BCI system, which recognizes the emotional states of users While they watch different music video clips. More precisely, the users are asked to watch several emotive music video clips, while their EEG and other peripheral physiological signals are acquired. Signal processing and machine learning algorithms are then developed to infer the emotional state of the user, induced while watching the music video clips. We propose the application of this BCI system in implicit emotional annotation of multimedia contents and sketch a iii iv road map towards developing a hybrid BCI system for emotional search and retrieval of multimedia contents. In the last topic, we present the results of our study on exploring the EEG alterations during the perception of pleasant and unpleasant odorant stimuli. We identify the regions of the brain cortex, which contribute to differentiation of pleasant and unpleasant odors and show that a relatively high accuracy for classification of EEG signals during perception of hedonically different odorant stimuli can be achieved. Furthermore, we identify a novel clinical application for the developed system, namely to study the EEG changes in comatose patients during stimulation with hedonically different odors in order to estimate the depth of coma.

Bayesian machine learning applied in a brain-computer interface for disabled users

Ulrich Hoffmann

A brain-computer interface (BCI) is a system that enables control of devices or communication with other persons, only through cerebral activity, without using muscles. The main application for BCIs is assistive technology for disabled persons. Examples for devices that can be controlled by BCIs are artificial limbs, spelling devices, or environment control systems. BCI research has seen renewed interest in recent years, and it has been convincingly shown that communication via a BCI is in principle feasible. However, present day systems still have shortcomings that prevent their widespread application. In part, these shortcomings are caused by limitations in the functionality of the pattern recognition algorithms used for discriminating brain signals in BCIs. Moreover, BCIs are often tested exclusively with able-bodied persons instead of conducting tests with the target user group, namely disabled persons. The goal of this thesis is to extend the functionality of pattern recognition algorithms for BCI systems and to move towards systems that are helpful for disabled users. We discuss extensions of linear discriminant analysis (LDA), which is a simple but efficient method for pattern recognition. In particular, a framework from Bayesian machine learning, the so-called evidence framework, is applied to LDA. An algorithm is obtained that learns classifiers quickly, robustly, and fully automatically. An extension of this algorithm allows to automatically reduce the number of sensors needed for acquisition of brain signals. More specifically, the algorithm allows to perform electrode selection. The algorithm for electrode selection is based on a concept known as automatic relevance determination (ARD) in Bayesian machine learning. The last part of the algorithmic development in this thesis concerns methods for computing accurate estimates of class probabilities in LDA-like classifiers. These probabilities are used to build a BCI that dynamically adapts the amount of acquired data, so that a preset, approximate bound on the probability of misclassifications is not exceeded. To test the algorithms described in this thesis, a BCI specifically tailored for disabled persons is introduced. The system uses electroencephalogram (EEG) signals and is based on the P300 evoked potential. Datasets recorded from five disabled and four able-bodied subjects are used to show that the Bayesian version of LDA outperforms plain LDA in terms of classification accuracy. Also, the impact of different static electrode configurations on classification accuracy is tested. In addition, experiments with the same datasets demonstrate that the algorithm for electrode selection is computationally efficient, yields physiologically plausible results, and improves classification accuracy over static electrode configurations. The classification accuracy is further improved by dynamically adapting the amount of acquired data. Besides the datasets recorded from disabled and able-bodied subjects, benchmark datasets from BCI competitions are used to show that the algorithms discussed in this thesis are competitive with state-of-the-art electroencephalogram (EEG) classification algorithms. While the experiments in this thesis are uniquely performed with P300 datasets, the presented algorithms might also be useful for other types of BCI systems based on the EEG. This is the case because functionalities such as robust and automatic computation of classifiers, electrode selection, and estimation of class probabilities are useful in many BCI systems. Seen from a more general point of view, many applications that rely on the classification of cerebral activity could possibly benefit from the methods developed in this thesis. Among the potential applications are interrogative polygraphy ("lie detection") and clinical applications, for example coma outcome prognosis and depth of anesthesia monitoring.

EPFL2007

Emotion Detection and Recognition based on Brain and Peripheral Physiological Signals

Eleni Kroupi

Emotions affect and determine social relationships and interactions, memory and creativity, and influence the mechanisms of rational thinking and decision making. The influence of emotion on decision making has gained attention in computer science. By detecting and recognizing emotions in an automatic way, machines endeavor to ease interaction between users and multimedia content. Automatic emotion detection and recognition can be carried out through analysis of users' various behavioural, and physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), electrodermal activity (EDA), and respiration, among others. These modalities have been extensively studied individually. However, since different individuals may experience the same emotion but express it differently, the various modalities are considered complementary, and fusion of various physiological and behavioural responses is expected to improve the quality of emotion recognition systems. Nevertheless, although representative features and their multimodal integration have been studied in affective computing research for various applications, patterns that arise from the dynamic interrelation among various modalities during emotional processes have received less attention. By summarizing each physiological signal only in a number of features, one may lose information present in the underlying dynamical co-evolution of various physiological signals. Considering all these issues, this thesis aims at detecting and recognizing emotion through brain and peripheral signals, targeting three complementary topics that have not been thoroughly explored. The first one includes emotion assessment from music video clips, the second one emotion assessment from odors, and the third one Quality of Experience (QoE) assessment from two-dimensional (2D) and three-dimensional (3D) video contents, using in all cases brain and peripheral physiological signals. Regarding emotion assessment from music video clips, subject-dependent and subject-independent analyses are carried out in this thesis, and the results reveal that although there are differentiations among the subjects' brain activation patterns, there are still common patterns across them. Moreover, the dynamical co-evolution between EEG and EDA is explored during emotional processes, and the results reveal that the coupling between EDA and EEG of the temporal lobe increases when strong emotions occur with respect to neutral ones. Finally, possible clustering patterns across subject-categories are investigated, and the results reveal that there are common characteristics across subject-categories related to their emotions. Regarding emotion assessment from odors, since the primary response to odors is related to pleasantness perception, which has not yet been thoroughly investigated, this thesis explores the way perceived odor pleasantness influences brain and periphery. In particular, two independent classifiers are trained and tested, one using EEG and the other using ECG features. The results reveal that it is possible to assess odor pleasantness perception from EEG and less accurately from ECG features, in a subject-independent framework. Also, decision fusion of the EEG and ECG classifiers is shown to discriminate odor pleasantness perception. Moreover, in order to explore the dynamical co-evolution between brain and peripheral signals, the coupling between heart rate and EEG is investigated. The results reveal that there is a significant increase in the coupling between ECG and temporal lobe EEG, when pleasant or unpleasant odors are experienced with respect to neutral ones. Enhanced QoE from multimedia contents targets to increase users' sensation of reality, in order to induce stronger emotions and render the user more involved in the experience. In this thesis, QoE is investigated in terms of four aspects, namely perceived depth, perceived overall quality, content preference, and sensation of reality. In particular, it is revealed that it is possible to recognize perceived depth, content preference, and sensation of reality from EEG signals, but not from the peripheral ones. Also, fusion between peripheral and EEG features is found to improve the performance in some cases. Finally, the left frontal cortex seems to be activated when sensation of reality is high, indicating that high sensation of reality is related to approach-related emotional processes. Although the three topics are independently explored in this thesis, their potential integration would endeavor to create immersive multimedia systems, which could adapt their properties according to users' emotions, and enhance, thus, user-experience.

EPFL2014

An efficient P300-based brain-computer interface for disabled subjects

Touradj Ebrahimi, Ulrich Hoffmann, Jean-Marc Vesin

A brain-computer interface (BCI) is a communication system that translates brain-activity into commands for a computer or other devices. In other words, a BCI allows users to act on their environment by using only brain-activity, without using peripheral nerves and muscles. In this paper, we present a BCI that achieves high classification accuracy and high bitrates for both disabled and able-bodied subjects. The system is based on the P300 evoked potential and is tested with five severely disabled and four able-bodied subjects. For four of the disabled subjects classification accuracies of 100% are obtained. The bitrates obtained for the disabled subjects range between 10 and 25 bits/min. The effect of different electrode configurations and machine learning algorithms on classification accuracy is tested. Further factors that are possibly important for obtaining good classification accuracy in P300-based BCI systems for disabled subjects are discussed.