Multimodal person recognition in audio-visual streams

Multimedia databases are growing rapidly in size in the digital age. To increase the value of these data and to enhance the user experience, there is a need to make these videos searchable through automatic indexing. Because people appearing and talking in the videos are often of high interest for end users, indices that represent the location and identity of people in the archive are indispensable for video search and browsing tools. On the other hand, multimedia videos contain resourceful data of people in both visual and auditory domains. This offers a potential for multimodal learning in the task of human identification. Hence, the main theme of this thesis is on algorithms to create indexes and exploit the audio-visual correspondence in large multimedia corpuses based on person identities.

First, this thesis deals with algorithms to create indexes through person discovery in videos. It involves several components: face and speaker diarization, face-voice association, and person naming. To obtain face clusters, we propose a novel face tracking approach that leverages face detectors with a tracking-by-detection framework relying on long term time-interval sensitive association costs. We use also shot context to further accelerate and improve face clustering. Face clusters are then associated to speaker clusters using dubbing and talking detection, in which a multimodal framework is introduced to represent the temporal relationship between the auditory and visual streams. We also improve speaker embeddings for recognition and clustering by using a regularizer called intra-class loss.

In the second half, the thesis focuses on multimodal learning with face-voice data. Here, we aim to answer two research questions. First, can one improve a voice embedding using knowledge transferred from a face representation? We investigate several transfer learning approaches to constrain the target voice embedding space to share latent attributes with the source face embedding space. The crossmodal constrains act as regularizers helping voice models, especially in the low-data setting. The second question is can face clusters be used as training labels to learn a speaker embedding? To answer this, we explore the tolerance of embedding losses under label uncertainty. From the risk minimization perspective, we obtain the analytical results that provide the heuristics in strategies to improve the tolerance against label noise. We apply the findings into our task of learning speaker embeddings using face clusters as labels. While the experimental results agree with the analytical heuristics, there is still a large gap in performance between the supervised and the weakly supervised models, which requires further investigation in the future.

Emotion Detection and Recognition based on Brain and Peripheral Physiological Signals

Eleni Kroupi

Emotions affect and determine social relationships and interactions, memory and creativity, and influence the mechanisms of rational thinking and decision making. The influence of emotion on decision making has gained attention in computer science. By detecting and recognizing emotions in an automatic way, machines endeavor to ease interaction between users and multimedia content. Automatic emotion detection and recognition can be carried out through analysis of users' various behavioural, and physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), electrodermal activity (EDA), and respiration, among others. These modalities have been extensively studied individually. However, since different individuals may experience the same emotion but express it differently, the various modalities are considered complementary, and fusion of various physiological and behavioural responses is expected to improve the quality of emotion recognition systems. Nevertheless, although representative features and their multimodal integration have been studied in affective computing research for various applications, patterns that arise from the dynamic interrelation among various modalities during emotional processes have received less attention. By summarizing each physiological signal only in a number of features, one may lose information present in the underlying dynamical co-evolution of various physiological signals. Considering all these issues, this thesis aims at detecting and recognizing emotion through brain and peripheral signals, targeting three complementary topics that have not been thoroughly explored. The first one includes emotion assessment from music video clips, the second one emotion assessment from odors, and the third one Quality of Experience (QoE) assessment from two-dimensional (2D) and three-dimensional (3D) video contents, using in all cases brain and peripheral physiological signals. Regarding emotion assessment from music video clips, subject-dependent and subject-independent analyses are carried out in this thesis, and the results reveal that although there are differentiations among the subjects' brain activation patterns, there are still common patterns across them. Moreover, the dynamical co-evolution between EEG and EDA is explored during emotional processes, and the results reveal that the coupling between EDA and EEG of the temporal lobe increases when strong emotions occur with respect to neutral ones. Finally, possible clustering patterns across subject-categories are investigated, and the results reveal that there are common characteristics across subject-categories related to their emotions. Regarding emotion assessment from odors, since the primary response to odors is related to pleasantness perception, which has not yet been thoroughly investigated, this thesis explores the way perceived odor pleasantness influences brain and periphery. In particular, two independent classifiers are trained and tested, one using EEG and the other using ECG features. The results reveal that it is possible to assess odor pleasantness perception from EEG and less accurately from ECG features, in a subject-independent framework. Also, decision fusion of the EEG and ECG classifiers is shown to discriminate odor pleasantness perception. Moreover, in order to explore the dynamical co-evolution between brain and peripheral signals, the coupling between heart rate and EEG is investigated. The results reveal that there is a significant increase in the coupling between ECG and temporal lobe EEG, when pleasant or unpleasant odors are experienced with respect to neutral ones. Enhanced QoE from multimedia contents targets to increase users' sensation of reality, in order to induce stronger emotions and render the user more involved in the experience. In this thesis, QoE is investigated in terms of four aspects, namely perceived depth, perceived overall quality, content preference, and sensation of reality. In particular, it is revealed that it is possible to recognize perceived depth, content preference, and sensation of reality from EEG signals, but not from the peripheral ones. Also, fusion between peripheral and EEG features is found to improve the performance in some cases. Finally, the left frontal cortex seems to be activated when sensation of reality is high, indicating that high sensation of reality is related to approach-related emotional processes. Although the three topics are independently explored in this thesis, their potential integration would endeavor to create immersive multimedia systems, which could adapt their properties according to users' emotions, and enhance, thus, user-experience.

EPFL2014

Social Network Analysis for Automatic Role Recognition

Sarah Favre

The computing community has shown a significant interest for the analysis of social interactions in the last decade. Different aspects of social interactions have been studied such as dominance, emotions, conflicts, etc. However, the recognition of roles has been neglected whereas these are a key aspect of social interactions. In fact, sociologists have shown not only that people play roles each time they interact, but also that roles shape behavior and expectations of interacting participants. The aim of this thesis is to fill this gap by investigating the problem of automatic role recognition in a wide range of interaction settings, including production environments, e.g. news and talk-shows, and spontaneous exchanges, e.g. meetings. The proposed role recognition approach includes two main steps. The first step aims at representing the individuals involved in an interaction with feature vectors accounting for their relationships with others. This step includes three main stages, namely segmentation of audio into turns (i.e. time intervals during which only one person talks), conversion of the sequence of turns into a social network, and use of the social network as a tool to extract features for each person. The second step uses machine learning methods to map the feature vectors into roles. The experiments have been carried out over roughly 90 hours of material. This is not only one of the largest databases ever used in literature on role recognition, but also the only one, to the best of our knowledge, including different interaction settings. In the experiments, the accuracy of the percentage of data correctly labeled in terms of roles is roughly 80% in production environments and 70% in spontaneous exchanges (lexical features have been added in the latter case). The importance of roles has been assessed in an application scenario as well. In particular, the thesis shows that roles help to segment talk-shows into stories, i.e. time intervals during which a single topic is discussed, with satisfactory performance. The main contributions of this thesis are as follows: To the best of our knowledge, this is the first work where social network analysis is applied to automatic analysis of conversation recordings. This thesis provides the first quantitative measure of how much roles constrain conversations, and a large corpus of recordings annotated in terms of roles. The results of this work have been published in one journal paper, and in five conference articles.

EPFL2011

Semi-supervised Adaptation of Assistant Based Speech Recognition Models for different Approach Areas

Petr Motlicek, Mittul Singh

Air Navigation Service Provider (ANSPs) replace paper flight strips through different digital solutions. The instructed commands from an air traffic controller (ATCOs) are then available in computer readable form. However, those systems require manual controller inputs, i.e. ATCOs’ workload increases. The Active Listening Assistant (AcListant®) project has shown that Assistant Based Speech Recognition (ABSR) is a potential solution to reduce this additional workload. However, the development of an ABSR application for a specific target-domain usually requires a large amount of manually transcribed audio data in order to achieve task- sufficient recognition accuracies. MALORCA project developed an initial basic ABSR system and semi-automatically tailored its recognition models for both Prague and Vienna approach by machine learning from automatically transcribed audio data. Command recognition error rates were reduced from 7.9% to under 0.6% for Prague and from 18.9% to 3.2% for Vienna.