A Probabilistic Temporal Model for Joint Attribute Extraction and Behavior Recognition

The focus of this paper is on the recognition of single object behavior from monocular image sequences. The general literature trend is to perform behavior recognition separately after an initial phase of feature/attribute extraction. We propose a framework where behavior recognition is performed jointly with attribute extraction, allowing the two tasks to mutually improve their results. To this end, we express the joint recognition / extraction problem in terms of a probabilistic temporal model, allowing its resolution via a variation of the Viterbi decoding algorithm, adapted to our model. Within the algorithm derivation, we translate probabilistic attribute extraction into a variational segmentation scheme. We demonstrate the viability of the proposed framework through a particular implementation for finger-spelling recognition. The obtained results illustrate the superiority of our collaborative model with respect to the traditional approach, where attribute extraction and behavior recognition are performed sequentially.

A collaborative approach to image segmentation and behavior recognition from image sequences

Laura Ioana Gui

Visual behavior recognition is currently a highly active research area. This is due both to the scientific challenge posed by the complexity of the task, and to the growing interest in its applications, such as automated visual surveillance, human-computer interaction, medical diagnosis or video indexing/retrieval. A large number of different approaches have been developed, whose complexity and underlying models depend on the goals of the particular application which is targeted. The general trend followed by these approaches is the separation of the behavior recognition task into two sequential processes. The first one is a feature extraction process, where features which are considered relevant for the recognition task are extracted from the input image sequence. The second one is the actual recognition process, where the extracted features are classified in terms of the pre-defined behavior classes. One problematic issue of such a two-pass procedure is that the recognition process is highly dependent on the feature extraction process, and does not have the possibility to influence it. Consequently, a failure of the feature extraction process may impair correct recognition. The focus of our thesis is on the recognition of single object behavior from monocular image sequences. We propose a general framework where feature extraction and behavior recognition are performed jointly, thereby allowing the two tasks to mutually improve their results through collaboration and sharing of existing knowledge. The intended collaboration is achieved by introducing a probabilistic temporal model based on a Hidden Markov Model (HMM). In our formulation, behavior is decomposed into a sequence of simple actions and each action is associated with a different probability of observing a particular set of object attributes within the image at a given time. Moreover, our model includes a probabilistic formulation of attribute (feature) extraction in terms of image segmentation. Contrary to existing approaches, segmentation is achieved by taking into account the relative probabilities of each action, which are provided by the underlying HMM. In this context, we solve the joint problem of attribute extraction and behavior recognition by developing a variation of the Viterbi decoding algorithm, adapted to our model. Within the algorithm derivation, we translate the probabilistic attribute extraction formulation into a variational segmentation model. The proposed model is defined as a combination of typical image- and contour-dependent energy terms with a term which encapsulates prior information, offered by the collaborating recognition process. This prior information is introduced by means of a competition between multiple prior terms, corresponding to the different action classes which may have generated the current image. As a result of our algorithm, the recognized behavior is represented as a succession of action classes corresponding to the images in the given sequence. Furthermore, we develop an extension of our general framework, that allows us to deal with a common situation encountered in applications. Namely, we treat the case where behavior is specified in terms of a discrete set of behavior types, made up of different successions of actions, which belong to a shared set of action classes. Therefore, the recognition of behavior requires the estimation of the most probable behavior type and of the corresponding most probable succession of action classes which explains the observed image sequence. To this end, we modify our initial model and develop a corresponding Viterbi decoding algorithm. Both our initial framework and its extension are defined in general terms, involving several free parameters which can be chosen so as to obtain suitable implementations for the targeted applications. In this thesis, we demonstrate the viability of the proposed framework by developing particular implementations for two applications. Both applications belong to the field of gesture recognition and concern finger-counting and finger-spelling. For the finger-counting application, we use our original framework, whereas for the finger-spelling application, we use its proposed extension. For both applications, we instantiate the free parameters of the respective frameworks with particular models and quantities. Then, we explain the training of the obtained models from specific training data. Finally, we present the results obtained by testing our trained models on new image sequences. The test results show the robustness of our models in difficult cases, including noisy images, occlusions of the gesturing hand and cluttered background. For the finger-spelling application, a comparison with the traditional sequential approach to image segmentation and behavior recognition illustrates the superiority of our collaborative model.

EPFL2008

Multimedia event modelling and recognition

Mark Barnard

The recognition of events in multimedia data is a challenging area of research. The growth in the amount of multimedia data being produced and stored increases the need for systems capable of automatically analysing this data. This analysis can aid in efficient browsing of and information retrieval from the data. The specific task addressed in this thesis is the recognition of sequences of high-level semantic events in both sports broadcasts and meeting data. In particular, the use of statistical machine learning techniques to model and recognise events in this task will be investigated. A vital task in multimedia event recognition is feature extraction. Feature extraction is an open problem in multimedia and currently there is no universal set of general robust features. In this thesis, we introduce a novel method of segmenting the playfield in sports. In this method statistical modelling, using Gaussian Mixture Models, is used to model the general grass playfield colour. Maximum a Posteriori adaptation is used to modify this general model to the specific colour of various grass sports fields. This method of playfield segmentation produced significantly improved results over existing methods. One issue in the task of multimedia event recognition is overfitting of statistical models when using low-level features and relatively small amounts of training data. One approach to this problem that is investigated here is the decomposition of the data into multiple streams based on different feature types. In this multi-stream event modelling each stream is processed separately and recognition decisions from each stream are then combined to give an overall decision. We present a study of various multi-stream modelling techniques, using variations of Hidden Markov Models (HMMs), these are tested on the task of event recognition in sports and meetings. Multi-stream techniques offered some improvement in performance through the ability to balance the contribution from each stream. A different approach to the problem of overfitting in statistical models is to use a hierarchical approach to recognition. In Layered-HMMs the recognition process is broken down into a number of layers where the recognition decision from low-level events is used as data in order to recognise high-level events. A technique combining unsupervised clustering and Layered-HMMs to recognise sequences of high-level events in the sport of rugby is presented. More precisely, unsupervised clustering is used to define a number of low-level sub-events. Then the first layer of the Layered-HMM produces recognition probabilities for these sub-events and the high-level events are modelled in terms of these probabilities. The Layered-HMM method was found to give more robust performance in high level event recognition than standard modelling techniques.

EPFL2005

Variational Methods for Human Modeling

Timur Bagautdinov

A large part of computer vision research is devoted to building models and algorithms aimed at understanding human appearance and behaviour from images and videos. Ultimately, we want to build automated systems that are at least as capable as people when it comes to interpreting humans. Most of the tasks that we want these systems to solve can be posed as a problem of inference in probabilistic models. Although probabilistic inference in general is a very hard problem of its own, there exists a very powerful class of inference algorithms, variational inference, which allows us to build efficient solutions for a wide range of problems. In this thesis, we consider a variety of computer vision problems targeted at modeling human appearance and behaviour, including detection, activity recognition, semantic segmentation and facial geometry modeling. For each of those problems, we develop novel methods that use variational inference to improve the capabilities of the existing systems. First, we introduce a novel method for detecting multiple potentially occluded people in depth images, which we call DPOM. Unlike many other approaches, our method does probabilistic reasoning jointly, and thus allows to propagate knowledge about one part of the image evidence to reason about the rest. This is particularly important in crowded scenes involving many people, since it helps to handle ambiguous situations resulting from severe occlusions. We demonstrate that our approach outperforms existing methods on multiple datasets. Second, we develop a new algorithm for variational inference that works for a large class of probabilistic models, which includes, among others, DPOM and some of the state-of-the-art models for semantic segmentation. We provide a formal proof that our method converges, and demonstrate experimentally that it brings better performance than the state-of-the-art on several real-world tasks, which include semantic segmentation and people detection. Importantly, we show that parallel variational inference in discrete random fields can be seen as a special case of proximal gradient descent, which allows us to benefit from many of the advances in gradient-based optimization. Third, we propose a unified framework for multi-human scene understanding which simultaneously solves three tasks: multi-person detection, individual action recognition and collective activity recognition. Within our framework, we introduce a novel multi-person detection scheme, which relies on variational inference and jointly refines detection hypotheses instead of relying on suboptimal post-processing. Ultimately, our model takes as an inputs a frame sequence and produces a comprehensive description of the scene. Finally, we experimentally demonstrate that our method brings better performance than the state-of-the-art. Fourth, we propose a new approach for learning facial geometry with deep probabilistic models and variational methods. Our model is based on a variational autoencoder with multiple sets of hidden variables, which are capturing various levels of deformations, ranging from global to local, high-frequency ones. We experimentally demonstrate the power of the model on a variety of fitting tasks. Our model is completely data-driven and can be learned from a relatively small number of individuals.

EPFL2018