Multi-stream ASR: An Oracle Perspective

Multi-stream based automatic speech recognition (ASR) systems outperform their single stream counterparts, specially in case of noisy speech. The main issues in multi-stream systems are: a) Find the feature streams carrying complementary information, and b) Combine the outputs of the classifiers trained on these feature streams. This paper investigates an `oracle' test to address these issues. An interpretation of the oracle test is proposed that can indicate the complimentary of the feature streams used in a multi-stream system. Also, we investigate the statistical nature of the oracle and study the relationship between oracle selection and entropy at the output of the classifiers. In the experiments carried out on two multi-stream systems, approximately 80% of the time oracle selected the classifier which had the minimum output entropy. The oracle analysis is supported by results obtained on multi-stream systems using different feature streams and inverse entropy method for weighting.