Multimedia event modelling and recognition

The recognition of events in multimedia data is a challenging area of research. The growth in the amount of multimedia data being produced and stored increases the need for systems capable of automatically analysing this data. This analysis can aid in efficient browsing of and information retrieval from the data. The specific task addressed in this thesis is the recognition of sequences of high-level semantic events in both sports broadcasts and meeting data. In particular, the use of statistical machine learning techniques to model and recognise events in this task will be investigated. A vital task in multimedia event recognition is feature extraction. Feature extraction is an open problem in multimedia and currently there is no universal set of general robust features. In this thesis, we introduce a novel method of segmenting the playfield in sports. In this method statistical modelling, using Gaussian Mixture Models, is used to model the general grass playfield colour. Maximum a Posteriori adaptation is used to modify this general model to the specific colour of various grass sports fields. This method of playfield segmentation produced significantly improved results over existing methods. One issue in the task of multimedia event recognition is overfitting of statistical models when using low-level features and relatively small amounts of training data. One approach to this problem that is investigated here is the decomposition of the data into multiple streams based on different feature types. In this multi-stream event modelling each stream is processed separately and recognition decisions from each stream are then combined to give an overall decision. We present a study of various multi-stream modelling techniques, using variations of Hidden Markov Models (HMMs), these are tested on the task of event recognition in sports and meetings. Multi-stream techniques offered some improvement in performance through the ability to balance the contribution from each stream. A different approach to the problem of overfitting in statistical models is to use a hierarchical approach to recognition. In Layered-HMMs the recognition process is broken down into a number of layers where the recognition decision from low-level events is used as data in order to recognise high-level events. A technique combining unsupervised clustering and Layered-HMMs to recognise sequences of high-level events in the sport of rugby is presented. More precisely, unsupervised clustering is used to define a number of low-level sub-events. Then the first layer of the Layered-HMM produces recognition probabilities for these sub-events and the high-level events are modelled in terms of these probabilities. The Layered-HMM method was found to give more robust performance in high level event recognition than standard modelling techniques.

Text detection and recognition in images and video sequences

Datong Chen

Text characters embedded in images and video sequences represents a rich source of information for content-based indexing and retrieval applications. However, these text characters are difficult to be detected and recognized due to their various sizes, grayscale values and complex backgrounds. This thesis investigates methods for building an efficient application system for detecting and recognizing text of any grayscale values embedded in images and video sequences. Both empirical image processing methods and statistical machine learning and modeling approaches are studied in two sub-problems: text detection and text recognition. Applying machine learning methods for text detection encounters difficulties due to character size, grayscale variations and heavy computation cost. To overcome these problems, we propose a two-step localization/verification approach. The first step aims at quickly localizing candidate text lines, enabling the normalization of characters into a unique size. In the verification step, a trained support vector machine or multi-layer perceptrons is applied on background independent features to remove the false alarms. Text recognition, even from the detected text lines, remains a challenging problem due to the variety of fonts, colors, the presence of complex backgrounds and the short length of the text strings. Two schemes are investigated addressing the text recognition problem: bi-modal enhancement scheme and multi-modal segmentation scheme. In the bi-modal scheme, we propose a set of filters to enhance the contrast of black and white characters and produce a better binarization before recognition. For more general cases, the text recognition is addressed by a text segmentation step followed by a traditional optical character recognition (OCR) algorithm within a multi-hypotheses framework. In the segmentation step, we model the distribution of grayscale values of pixels using a Gaussian mixture model or a Markov Random Field. The resulting multiple segmentation hypotheses are post-processed by a connected component analysis and a grayscale consistency constraint algorithm. Finally, they are processed by an OCR software. A selection algorithm based on language modeling and OCR statistics chooses the text result from all the produced text strings. Additionally, methods for using temporal information of video text are investigated. A Monte Carlo video text segmentation method is proposed for adapting the segmentation parameters along temporal text frames. Furthermore, a ROVER (Recognizer Output Voting Error Reduction) algorithm is studied for improving the final recognition text string by voting the characters through temporal frames.

2003

Text detection and recognition in images and video sequences

Datong Chen

EPFL2003

Automatic social role recognition and its application in structuring multiparty interactions

Ashtosh Sapru

Automatic processing of multiparty interactions is a research domain with important applications in content browsing, summarization and information retrieval. In recent years, several works have been devoted to find regular patterns which speakers exhibit in a multiparty interaction also known as social roles. Most of the research in literature has generally focused on recognition of scenario specific formal roles. More recently, role coding schemes based on informal social roles have been proposed in literature, defining roles based on the behavior speakers have in the functioning of a small group interaction. Informal social roles represent a flexible classification scheme that can generalize across different scenarios of multiparty interaction. In this thesis, we focus on automatic recognition of informal social roles and exploit the influence of informal social roles on speaker behavior for structuring multiparty interactions. To model speaker behavior, we systematically explore various verbal and non verbal cues extracted from turn taking patterns, vocal expression and linguistic style. The influence of social roles on the behavior cues exhibited by a speaker is modeled using a discriminative approach based on conditional random fields. Experiments performed on several hours of meeting data reveal that classification using conditional random fields improves the role recognition performance. We demonstrate the effectiveness of our approach by evaluating it on previously unseen scenarios of multiparty interaction. Furthermore, we also consider whether formal roles and informal roles can be automatically predicted by the same verbal and nonverbal features. We exploit the influence of social roles on turn taking patterns to improve speaker diarization under distant microphone condition. Our work extends the Hidden Markov model (HMM)- Gaussian mixture model (GMM) speaker diarization system, and is based on jointly estimating both the speaker segmentation and social roles in an audio recording. We modify the minimum duration constraint in HMM-GMM diarization system by using role information to model the expected duration of speaker's turn. We also use social role n-grams as prior information to model speaker interaction patterns. Finally, we demonstrate the application of social roles for the problem of topic segmentation in meetings. We exploit our findings that social roles can dynamically change in conversations and use this information to predict topic changes in meetings. We also present an unsupervised method for topic segmentation which combines social roles and lexical cohesion. Experimental results show that social roles improve performance of both speaker diarization and topic segmentation.

EPFL2015

Text detection and recognition in images and video sequences

Datong Chen

EPFL2003

Text detection and recognition in images and video sequences

Datong Chen

2003

Automatic social role recognition and its application in structuring multiparty interactions

Ashtosh Sapru

EPFL2015