Exploiting Recognition of Natural Visual Stimuli for Brain Computer Interfacing

Visual search is a good illustration about how the brain coordinates a variety of functions such as visual clues extraction from the scene, coordination of the eye-movements, accumu- lation of visual information and visual recognition. It has an important role in many daily life activities of people – the search of a familiar face in a crowd, a book in the library, or a building in a street. In all these scenarios, it represents a recognition of a specific target object in the presence of predominant task-irrelevant visual information. This thesis specifically addresses the visual search in context of brain-computer interaction (BCI) where the accent is on visually rich stimulation, approaching the real-world BCI applications. Different cognitive aspects of visual processing have been extensively studied using the Rapid Serial Visual Presentation (RSVP) protocol – a sequentially presentation of visual stimuli at the same location on a display. The main part of this thesis deals with the visual search under the RSVP protocol. In such a protocol, the electroencephalogram (EEG) reflects whether a subject recognize an image as a target. Our focus is on extracting and exploiting this informa- tion for the BCI application of image retrieval. We designed a framework for EEG-based image search which involves coupling of the EEG decoder with the computer vision technique for image retrieval. These two components of the system are seen as complementary – bringing in the semantic information and large scale computing, respectively. Our design is founded on an iterative coupling, which relaxes a performance requirement of the individual components. We report the results of the experimental evaluation, where natural color images were used as stimuli. This thesis also includes a comprehensive analysis of the EEG correlates of visual recogni- tion in the EEG-based image search task, addressing the effects of the less controllable stimuli to the EEG potentials’ attributes. To this end, our focus is on the spatio-temporal aspects of the EEG potentials, along with the modulation of the EEG-based cortical connectivity estimates. Finally, this thesis presents an exploratory research on active visual search in the context of BCI. An essential part of active searching of the visual scene represents eye movements. Hence, this type of task better reflects our natural behavior and as such extends the possibilities of BCI applications. We designed two experimental protocols to study the active visual search – guided and free-view visual search. We report the results of the combined study of eye tracking data and simultaneously recorded EEG.

Towards everyday brain-computer interfacing through decoding visual cognition

Ruslan Aydarkhanov

Decoding visual cognition from non-invasive measurements of brain activity has shown valuable applications. Vision-based Brain-Computer Interfaces (BCI) systems extend from spellers to database search and spatial navigation. Despite the high performance of these systems, they are limited to the laboratory environment. Several attempts were made to improve the quality of life of patients with severe neurodegenerative disorders by helping them acquire more independence. Bringing vision-based BCI to everyday life would benefit a wide range of population from handicapped to healthy people. However, real-life applications impose additional challenges on obtaining neural signals with a good signal-to-noise ratio. The natural environments are rich, dynamic and ambiguous that challenges the visual perception and, therefore, the decoding of underlying neural correlates. The limited attentional resources require to explore and visually sample the environment by freely moving eyes and fixating on the relevant objects. Moreover, everyday activities are rarely isolated leading to a mixture and complex interactions between neural correlates in the measured signal. In this thesis, I explore various aspects of decoding brain activity by developing new protocols for realistic scenarios with challenging visual tasks. The tasks include free visual exploration, dynamic scenes, and multiple tasks, for example, recognition of a billboard category during simulated driving. The major efforts have been directed towards synchronous decoding of Event-Related Potentials (ERP) acquired by the means of electroencephalography (EEG) and Eye-tracking technology, which allowed the extraction of Eye-Fixation Related Potentials (EFRP). Additionally, I investigate the neural correlates of perceptual decision making which is tightly linked to visual recognition. The perceptual challenges can lead to higher temporal variability of obtained ERP which was also addressed from the decoding perspective. The obtained results, firstly, provide new insights on how to tackle the temporal variability of ERP. Secondly, they raise new questions on studying and decoding neural correlates of perceptual decision making. And thirdly, they show the feasibility of decoding visual recognition in a simulated realistic scenario of car driving. Although, as expected, I found a lower performance compared to classical ERP protocols, these findings hold promise and raise new questions to be investigated in order to improve the quality of decoding visual cognition for everyday application.

EPFL2020

Learning the structure of image collections with latent aspect models

Florent Monay Michaud

The approach to indexing an image collection depends on the type of data to organize. Satellite images are likely to be searched with latitude and longitude coordinates, medical images are often searched with an image example that serves as a visual query, and personal image collections are generally browsed by event. A more general retrieval scenario is based on the use of textual keywords to search for images containing a specific object, or representing a given scene type. This requires the manual annotation of each image in the collection to allow for the retrieval of relevant visual information based on a text query. This time-consuming and subjective process is the current price to pay for a reliable and convenient text-based image search. This dissertation investigates the use of probabilistic models to assist the automatic organization of image collections, attempting to link the visual content of digital images with a potential textual description. Relying on robust, patch-based image representations that have proven to capture a variety of visual content, our work proposes to model images as mixtures of latent aspects. These latent aspects are defined by multinomial distributions that capture patch co-occurrence information observed in the collection. An image is not represented by the direct count of its constituting elements, but as a mixture of latent aspects that can be estimated with principled, generative unsupervised learning methods. An aspect-based image representation therefore incorporates contextual information from the whole collection that can be exploited. This emerging concept is explored for several fundamental tasks related to image retrieval – namely classification, clustering, segmentation, and annotation – in what represents one of the first coherent and comprehensive study of the subject. We first investigate the possibility of classifying images based on their estimated aspect mixture weights, interpreting latent aspect modeling as an unsupervised feature extraction process. Several image categorization tasks are considered, where images are classified based on the present objects or according to their global scene type. We demonstrate that the concept of latent aspects allows to take advantage of non-labeled data to infer a robust image representation that achieves a higher classification performance than the original patch-based representation. Secondly, further exploring the concept, we show that aspects can correspond to an interesting soft clustering of an image collection that can serve as a browsing structure. Images can be ranked given an aspect, illustrating the corresponding co-occurrence context visually. In the third place, we derive a principled method that relies on latent aspects to classify image patches into different categories. This produces an image segmentation based on the resulting spatial class-densities. We finally propose to model images and their caption with a single aspect model, merging the co-occurrence contexts of the visual and the textual modalities in different ways. Once a model has been learned, the distribution of words given an unseen image is inferred based on its visual representation, and serves as textual indexing. Overall, we demonstrate with extensive experiments that the co-occurrence context captured by latent aspects is suitable for the above mentioned tasks, making it a promising approach for multimedia indexing.

EPFL2007

EEG-based Recognition of Covert Attentional Shifts

Catherine Brandner, Ricardo Andres Chavarriaga Lozano

Attentional shifts precede most of the perceptual processes, and therefore their recognition may constitute an interesting process for both the study of human perception as well as the design of EEG-based Brain Computer Interfaces (BCI). Several studies have focused on EEG modulations of spatial attention, pointing out to changes in both Alpha and Gamma bands (Thut et al. 2006 J. Neurosci 26:9494; Tallon-Baudry and Bertrand 1999 T Cogn Sci 3:151; Jensen et al. 2007 TINS 30:317). This work aims at further characterization of EEG correlates of attention shifts to evaluate their potential use in BCI applications. We recorded 32-channel scalp EEG while subjects covertly shift their attention towards one out of several spatial targets. Consistently with previous studies, characterization of the spatio-temporal patterns using time-frequency analysis (Morlet wavelet transformation), shows main modulations in the alpha and gamma frequency bands. Moreover, it is also shown that these modulations are characterized by intermittent episodes of activity (Palix et al. 2006 Perception 25:234). Furthermore, we assess the EEG-based recognition of spatial attention -time and target location of the shift- using a machine learning approach based on the detection of informative, episodic activity. The proposed approach is based on the hypothesis that recognition may be improved by identifying the moments where the underlying neural phenomena is more salient (e. g. the appearance of episodic oscillations related to attentional shifts), and performing the classification based mainly -if not exclusively- on the activity during these periods. Our results show accurate recognition of attentional shifts (>75%) based on the scalp EEG signal. Moreover, the proposed approach allow us to efficiently detect endogenous processes loosely synchronized to external events. These results provides evidence supporting the feasibility of using attentional processes in Brain-Computer Interfaces.

2008