Enabling Speech Applications using Ad Hoc Microphone Arrays

Microphone arrays are central players in hands-free speech interface applications. The main duty of a microphone array is capturing distant-talking speech with high quality. A microphone array can acquire the desired speech signals selectively by leading the beampattern towards the desired speaker. The foreseen application of ubiquitous sensing motivated by the abundance of microphone-embedded devices, such as notebooks and smart phones, raises the importance of research on ad~hoc microphone arrays. The key challenges pertain to the unknown geometry of the microphones and asynchronous recordings. The goal of this PhD thesis is to address the issues of microphone and source localization to enable beamforming for higher level speech processing tasks. To that end, we exploit the prior knowledge of the acoustical and geometrical structures underlying the ad~hoc distributed nodes to devise novel algorithms for microphone array calibration and source localization, as well as beamforming techniques for distant speech applications. To address the problem of ad~hoc microphone array calibration, the analytic diffuse sound field coherence model is investigated and its fundamental properties are studied. This model enables pairwise distance estimation for calibration of a relatively compact microphone array. We derive the mathematical framework for estimation of long pairwise distances exploiting the low-rank properties of the Euclidean distance matrix and develop a novel matrix completion algorithm for ad hoc microphone array calibration along with theoretical guarantees. Furthermore, the problem of source localization using ad~hoc microphones in a reverberant enclosure is addressed. We incorporate the image model of multipath propagation for construction of a Euclidean distance matrix. The low-rank structure of the distance matrix is exploited to identify the support of the room impulse response function and its unique map to the source location. This approach enables single-channel and distributed source localization from asynchronous recordings provided by ad hoc microphones. Along this line, we address the problem of robust microphone array placement to optimize the localization performance. Finally, spatial filtering techniques relying on beamforming are investigated for high quality speech acquisition and higher level applications. We develop beamformers for joint multi-speaker localization and voice activity detection. In addition, the broadband beampattern of a microphone array is characterized and its relation to predict the speech recognition accuracy is desired.

COMBINING CEPSTRAL NORMALIZATION AND COCHLEAR IMPLANT-LIKE SPEECH PROCESSING FOR MICROPHONE ARRAY-BASED SPEECH RECOGNITION

Philip Neil Garner, Mohammadjavad Taghizadeh

This paper investigates the combination of cepstral normalization and cochlear implant-like speech processing for microphone array- based speech recognition. Testing speech signals are recorded by a circular microphone array and are subsequently processed with superdirective beamforming and McCowan post-filtering. Training speech signals, from the multichannel overlapping Number corpus (MONC), are clean and not overlapping. Cochlear implant-like speech processing, which is inspired from the speech processing strategy in cochlear implants, is applied on the training and testing speech signals. Cepstral normalization, including cepstral mean and variance normalization (CMN and CVN), are applied on the training and testing cepstra. Experiments show that implementing either cepstral normalization or cochlear implant-like speech pro- cessing helps in reducing the WERs of microphone array-based speech recognition. Combining cepstral normalization and cochlear implant-like speech processing reduces further the WERs, when there is overlapping speech. Train/test mismatches are measured using the Kullback-Leibler divergence (KLD), between the global probability density functions (PDFs) of training and testing cepstral vectors. This measure reveals a train/test mismatch reduction when either cepstral normalization or cochlear implant-like speech pro- cessing is used. It reveals also that combining these two processing reduces further the train/test mismatches as well as the WERs.

2012

Enhanced Phone Posteriors for Improving Speech Recognition Systems

Hervé Bourlard, Hamed Ketabdar

Using phone posterior probabilities has been increasingly explored for improving automatic speech recognition (ASR) systems. In this paper, we propose two approaches for hierarchically enhancing these phone posteriors, by integrating long acoustic context, as well as prior phonetic and lexical knowledge. In the first approach, phone posteriors estimated with a Multi-Layer Perceptron (MLP), are used as emission probabilities in HMM forward-backward recursions. This yields new enhanced posterior estimates integrating HMM topological constraints (encoding specific phonetic and lexical knowledge), and context. posteriors are post-processed by a secondary MLP, in order to learn inter and intra dependencies between the phone posteriors. These dependencies are prior phonetic knowledge. The learned knowledge is integrated in the posterior estimation during the inference (forward pass) of the second MLP, resulting in enhanced phone posteriors. We investigate the use of the enhanced posteriors in hybrid HMM/ANN and Tandem configurations. We propose using the enhanced posteriors as replacement, or as complementary evidences to the regular MLP posteriors. The proposed method has been tested on different small and large vocabulary databases, always resulting in consistent improvements in frame, phone and word recognition rates.

IDIAP2008

Verified Speaker Localization Utilizing Voicing Level in Split-bands

Afsaneh Asaei, Mohammadjavad Taghizadeh

This paper proposes a joint verification-localization structure based on split-band analysis of speech signal and the mixed voicing level. To address the problems in reverberant acoustic environments, a new fundamental frequency estimation algorithm is proposed based on high resolution spectral estimation. In the reconstruction of the distorted speech this information is utilized to reduce the side effect of acoustic noise on the voicing parts. A speaker verification system examines the features of the reconstructed speech in order to authorize the speaker before localization. This procedure prevents localization and beamforming for non-speech and specially the unwanted speakers in multi-speaker scenarios. The verification is implemented with the Gaussian Mixture Model and a new filtering scheme is proposed based on the voicing likelihood of each frequency band measured in the previous steps for efficient localization of the authorized speaker. The performance of the proposed VSL (verified speaker localization) front-end is evaluated in various reverberant and noisy environments. The VSL is utilized in the development of distant-talking automatic speech recognition by microphone array where the system can lock on a specific source and hence the recognition quality improves noticeably.

2009