Impact of Sample Sizes on Information Theoretic Measures for Audio-Visual Signal Processing

In this paper we aim to explore what is the most appropriate number of data samples needed when measuring the temporal correspondence between a chosen set of video and audio cues in a given audio-visual sequence. Presently the optimal model that connects statistics of audio and video signals does not exist since one does not know the most appropriate features to be extracted in order to analyze their correlation. Previous approaches assumed simple parametric and non-parametric models for the joint distribution for capturing the complex signal relationships. The main problem in using the standard information theoretic quantities, such as entropy and mutual information, is the accurate estimation of the probability density function from a limited number of data. The main idea is to project the data into a statistically sufficient low-dimensional subspace, suitable for density estimation. Then using a simple parametric model based on assumption of Gaussianity, mutual information is estimated and applied as a measure of correspondence. We exploit how the choice of sample size affects the reliability of the correspondence measure (mutual information) between selected features of the two modalities, audio and video.

Extraction of Audio Features Specific to Speech using Information Theory and Differential Evolution

Patricia Besson, Murat Kunt, Vlad Popovici, Jean-Philippe Thiran, Jean-Marc Vesin

We present a method that exploits an information theoretic framework to extract optimized audio features using the video information. A simple measure of mutual information (MI) between the resulting audio features and the video ones allows to detect the active speaker among different candidates. Our method involves the optimization of an MI-based objective function. No approximation is introduced to solve this optimization problem, neither concerning the estimation of the probability density functions (pdf) of the features, nor the cost function itself. The pdf are estimated from the samples using a non-parametric approach. As far as concern the optimization process itself, three different optimization methods (one local and two globals) are compared in this paper. The Differential Evolution algorithm is shown to be outstanding performant for our problem and is threrefore eventually retains. Two information theoretic optimization criteria are compared and their ability to extract audio features specific to speeh is discussed. As a result, our method achieves a speaker detection rate of 100% on our test sequences, and of 95% on a state-of-the-art sequence.

2005

Extraction of audio features specific to speech production for multimodal speaker detection

Patricia Besson, Murat Kunt, Vlad Popovici, Jean-Philippe Thiran, Jean-Marc Vesin

A method that exploits an information theoretic framework to extract optimized audio features using video information is presented. A simple measure of mutual information (MI) between the resulting audio and video features allows the detection of the active speaker among different candidates. This method involves the optimization of an MI-based objective function. No approximation is needed to solve this optimization problem, neither for the estimation of the probability density functions (pdf) of the features, nor for the cost function itself. The pdf are estimated from the samples using a non-parametric approach. The challenging optimization problem is solved using a global method: the Differential Evolution algorithm. Two information theoretic optimization criteria are compared and their ability to extract audio features specific to speech is discussed. Using these specific speech audio features, candidates video features are then classified as membership of the "speaker" or "non-speaker" class, resulting in a speaker detection scheme. As a result, our method achieves a speaker detection rate of 100% on home- grown test sequences, and of 85% on most commonly used sequences.

2008

Statistical limits of high-dimensional inference problems

Clément Dominique Luneau

This thesis focuses on two kinds of statistical inference problems in signal processing and data science. The first problem is the estimation of a structured informative tensor from the observation of a noisy tensor in which it is buried. The structure comes from the possibility to decompose the informative tensor as the sum of a small number of rank-one tensors (small compared to its size). Such structure has applications in data science where data, organized into arrays, can often be explained by the interaction between a few features characteristic of the problem. The second problem is the estimation of a signal input to a feedforward neural network whose output is observed. It is relevant for many applications (phase retrieval, quantized signals) where the relation between the measurements and the quantities of interest is not linear. We look at these two statistical models in different high-dimensional limits corresponding to situations where the amount of observations and size of the signal become infinitely large. These asymptotic regimes are motivated by the ever-increasing computational power and storage capacity that make possible the processing and handling of large data sets. We take an information-theoretic approach in order to establish fundamental statistical limits of estimation in high-dimensional regimes. In particular, the main contributions of this thesis are the proofs of exact formulas for the asymptotic normalized mutual information associated with these inference problems. These are low-dimensional variational formulas that can nonetheless capture the behavior of a large system where each component interacts with all the others. Owing to the relationship between mutual information and Bayesian inference, we use the solutions to these variational problems to rigorously predict the asymptotic minimum mean-square error (MMSE), the error achieved by the (Bayes) optimal estimator. We can thus compare algorithmic performances to the statistical limit given by the MMSE. Variational formulas for the mutual information are referred to as replica symmetric (RS) ansätze due to the predictions of the heuristic replica method from statistical physics. In the past decade proofs of the validity of these predictions started to emerge. The general strategy is to show that the RS ansatz is both an upper and lower bound on the asymptotic normalized mutual information. The present work leverages on the adaptive interpolation method that proposes a unified way to prove the two bounds. We extend the adaptive interpolation to situations where the order parameter of the problem is not a scalar but a matrix, and to high-dimensional regimes that differ from the one for which the RS formula is usually conjectured. Our proofs also demonstrate the modularity of the method. Indeed, using statistical models previously studied in the literature as building blocks of more complex ones (e.g., estimated signal with a model of structure), we derive RS formulas for the normalized mutual information associated with estimation problems that are relevant to modern applications.

EPFL2021

Statistical limits of high-dimensional inference problems

Clément Dominique Luneau

EPFL2021