Mutual information eigenlips for audio-visual speech recognition

This paper proposes an application of information theoretic approach for finding the most informative subset of eigenfeatures to be used for audio-visual speech recognition tasks. The state-of-the-art visual feature extraction methods in the area of speechreading rely on either pixel or geometric based methods or their combination. However, there is no common rule defining how these features have to be selected with respect to the chosen set of audio cues and how well they represent the classes of the uttered speech. Our main objective is to exploit the complementarity of audio and visual sources and select meaningful visual descriptors by the means of mutual information. We focus on the principal components projections of the mouth region images and apply the proposed method such that only those cues having the highest mutual information with word classes are retained. The algorithm is tested by performing various speech recognition experiments on a chosen audio-visual dataset. The obtained recognition rates are compared to those acquired using a conventional principal component analysis and promising results are shown.

High-Dimensional Inference on Dense Graphs with Applications to Coding Theory and Machine Learning

Mohamad Baker Dia

We are living in the era of "Big Data", an era characterized by a voluminous amount of available data. Such amount is mainly due to the continuing advances in the computational capabilities for capturing, storing, transmitting and processing data. However, it is not always the volume of data that matters, but rather the "relevant" information that resides in it. Exactly 70 years ago, Claude Shannon, the father of information theory, was able to quantify the amount of information in a communication scenario based on a probabilistic model of the data. It turns out that Shannon's theory can be adapted to various probability-based information processing fields, ranging from coding theory to machine learning. The computation of some information theoretic quantities, such as the mutual information, can help in setting fundamental limits and devising more efficient algorithms for many inference problems. This thesis deals with two different, yet intimately related, inference problems in the fields of coding theory and machine learning. We use Bayesian probabilistic formulations for both problems, and we analyse them in the asymptotic high-dimensional regime. The goal of our analysis is to assess the algorithmic performance on the first hand and to predict the Bayes-optimal performance on the second hand, using an information theoretic approach. To this end, we employ powerful analytical tools from statistical physics. The first problem is a recent forward-error-correction code called sparse superposition code. We consider the extension of such code to a large class of noisy channels by exploiting the similarity with the compressed sensing paradigm. Moreover, we show the amenability of sparse superposition codes to perform joint distribution matching and channel coding. In the second problem, we study symmetric rank-one matrix factorization, a prominent model in machine learning and statistics with many applications ranging from community detection to sparse principal component analysis. We provide an explicit expression for the normalized mutual information and the minimum mean-square error of this model in the asymptotic limit. This allows us to prove the optimality of a certain iterative algorithm on a large set of parameters. A common feature of the two problems stems from the fact that both of them are represented on dense graphical models. Hence, similar message-passing algorithms and analysis tools can be adopted. Furthermore, spatial coupling, a new technique introduced in the context of low-density parity-check (LDPC) codes, can be applied to both problems. Spatial coupling is used in this thesis as a "construction technique" to boost the algorithmic performance and as a "proof technique" to compute some information theoretic quantities. Moreover, both of our problems retain close connections with spin glass models studied in statistical mechanics of disordered systems. This allows us to use sophisticated techniques developed in statistical physics. In this thesis, we use the potential function predicted by the replica method in order to prove the threshold saturation phenomenon associated with spatially coupled models. Moreover, one of the main contributions of this thesis is proving that the predictions given by the "heuristic" replica method are exact. Hence, our results could be of great interest for the statistical physics community as well, as they help to set a rigorous mathematical foundation of the replica predictions.

EPFL2018

Variational Information Maximization in Gaussian Channels

Recently, we introduced a simple variational bound on mutual information, that resolves some of the difficulties in the application of information theory to machine learning. Here we study a specific application to Gaussian channels. It is well known that PCA may be viewed as the solution to maximizing information transmission between a high dimensional vector and its low dimensional representation . However, such results are based on assumptions of Gaussianity of the sources. In this paper, we show how our mutual information bound, when applied to this arena, gives PCA solutions, without the need for the Gaussian assumption. Furthermore, it naturally generalizes to providing an objective function for Kernel PCA, enabling the principled selection of kernel parameters.

IDIAP2004

Visual feature analysis for audio-visual speech recognition

Ivana Arsic de Heras Ciechomska

Humans perceive their surrounding environment in a multimodal manner by using multi-sensory inputs combined in a coordinated way. Various studies in psychology and cognitive science indicate the multimodal nature of human speech production and perception. Such findings have motivated the development of a new and fast growing research field of audio-visual speech processing. Exhaustive research done so far shows that systems using both visual and acoustic data do perform better than their conventional "blind" counterparts. The choice of suitable visual cues is of paramount importance in such systems. The aim of this thesis is to analyze visual features and evaluate their relevance from an automatic speech recognition point of view. Speech pertinent information is mainly conveyed in the mouth area and a fundamental requirement is to obtain its representation in a parameterized form. The first objective of this work is to address the problem of visual front end design. This represents a challenging issue that involves complex computer vision tasks of face detection and facial feature tracking. After finding the speaker's face, it is important to accurately detect the mouth area and subsequently track it throughout a video sequence. In this thesis a method for nose and mouth region of interest tracking, has been developed for the application scenario restricted to audio-visual speech recognition. Nose tracking is accomplished by employing a template matching technique. Further on, the mouth region is localized taking into account knowledge of geometric properties of the human face. The next objective of this work is set upon the need for robust and accurate extraction of key lip points of interest. To accomplish such a task two novel algorithms are developed. The first one aims to find four key lip points that enable normalization of mouth images. The second algorithm provides additional mouth shape descriptors and the basis for fitting labial contours. The emphasis of this study is on audio-visual speech recognition which requires visual features suitable for classification. A question that immediately arises is what the most speech salient features are. To date different methods are proposed that mainly rely on some a priori defined rules. Area based features are selected based on criteria inherited from image compression, where geometric based methods use set of cues selected as meaningfull one for humans. There is no guarantee that such representations are optimal for automatic recognition of the uttered speech. The study done in this thesis aims to observe this problem from a different angle by including basic information theoretic concepts in visual feature analysis. The selection criteria employs Mutual Information as a measure of how informative visual cues are. The concept of mutual information eigenlips is introduced to reflect the most informative eigenfeatures. Visual features obtained using an information theoretic framework are further used for audio-visual speech recognition experiments. The obtained results indicate improved system performance when such visual cues are applied. In conclusion, the proposed information theoretic framework for visual feature selection can be applied in other audio-visual speech processing areas. The possible applications include audio-visual speaker detection and speaker localization.