HMM modeling for audio-visual speech recognition

Abstract

Bimodal speech recognition is a robust technique for automated speech analysis, and has received a lot of attention in the last few decades. In this paper, we analyze the effect of the HMM models on the performance of the bimodal speech recognizer, present a comparative analysis of the different HMM models that can be used in bimodal speech recognition, and finally propose a novel model, which has been experimentally verified to perform better than others. One of the unique characteristic of our HMM model is the novel fusion strategy of the acoustic and the visual features, that takes into account the different sampling rates of these two signals. Compared to audio only, the bimodal speech recognition scheme has a much more improved recognition accuracy, especially in presence of noise.