Brain-Computer Interface in Multimedia Systems

A brain-computer interface (BCI) is a system that allows a user to communicate with the environment only through cerebral activity, without using muscular output channels. To establish a direct link between the brain and a computer, the electroencephalogram (EEG) signal of a user is measured and then analyzed with the help of signal processing and machine learning algorithms. Once a certain mental activity has been detected by the computer, a response can be displayed on a screen or a command can be sent to a peripheral device, for example a wheelchair or a television. The main application area for BCI is assistive technology for handicapped people. For example, one can imagine artificial limbs controlled by a BCI, a BCI-based spelling device, or an environment control system based on a BCI. Nevertheless, during the last years it has been convincingly shown that communication via a BCI is feasible for able-bodied as well as handicapped users and several new applications such as entertainment and gaming, and neuro-feedback have been proposed for BCI systems. This thesis describes the author’s contributions to the research on brain computer interface systems, focusing on exploring and studying new applications of BCIs especially in multimedia communication domain. After providing a general overview of the EEG signal processing techniques and current BCI systems, three topics of interest have been identified. These topics include BCI for salient image retrieval and image triage, BCI for affect recognition during multimedia consumption, and BCI for studying the EEG correlates of pleasant and unpleasant odors. In the first topic, we present a BCI system, capable of identifying interesting images in an image database. Using this system, images of a database are presented to several users at a fast rate, and the information on whether a given image is salient, is implicitly extracted by means of processing the EEG signals, acquired when users are watching the image sequence. Furthermore, in order to investigate the impact of expertise on a BCI-based salient image retrieval system, the changes in the EEG signals of expert and novice users (with respect to image content) are studied. We show that it is possible to define experimental protocol and judiciously apply signal processing tools such that a practical BCI system can be set up for detection of interest in users during the watching of image sequences. Furthermore, we show that a relatively high retrieval accuracy can be obtained for most of the users of this system. The second presented topic deals with an affective BCI system, which recognizes the emotional states of users While they watch different music video clips. More precisely, the users are asked to watch several emotive music video clips, while their EEG and other peripheral physiological signals are acquired. Signal processing and machine learning algorithms are then developed to infer the emotional state of the user, induced while watching the music video clips. We propose the application of this BCI system in implicit emotional annotation of multimedia contents and sketch a iii iv road map towards developing a hybrid BCI system for emotional search and retrieval of multimedia contents. In the last topic, we present the results of our study on exploring the EEG alterations during the perception of pleasant and unpleasant odorant stimuli. We identify the regions of the brain cortex, which contribute to differentiation of pleasant and unpleasant odors and show that a relatively high accuracy for classification of EEG signals during perception of hedonically different odorant stimuli can be achieved. Furthermore, we identify a novel clinical application for the developed system, namely to study the EEG changes in comatose patients during stimulation with hedonically different odors in order to estimate the depth of coma.