Visual speech recognition using active shape models and hidden Markov models

Abstract

This paper describes a novel approach for visual speech recognition. The shape of the mouth is modelled by an active shape model which is derived from the statistics of a training set and used to locate, track and parameterise the speaker's lip movements. The extracted parameters representing the lip shape are modelled as continuous probability distributions and their temporal dependencies are modelled by hidden Markov models. We present recognition tests performed on a database of a broad variety of speakers and illumination conditions. The system achieved an accuracy of 85.42% for a speaker independent recognition task of the first four digits using lip shape information only.