Tracking facial features under occlusions and recognizing facial expressions in sign language

Abstract

This paper presents work towards recognizing facial expressions that are used in sign language recognition. Facial features are tracked to effectively capture temporal visual cues on the signer's face during signing. A Bayesian framework is proposed as a feedback mechanism to the Kanade-Lucas-Tomasi (KLT) tracker for reliably tracking facial features in the presence of head motions and temporary occlusions by hand. This mechanism relies on a set of face shape subspaces learned by probabilistic principal component analysis with an update scheme to adapt to persons with different face shapes. The results show that the proposed tracker can track facial features with large head motions, substantial facial deformations, and temporary facial occlusions by hand. The tracked results were input to a recognition system comprising HMMs and a NN to recognize four common American sign language facial expressions.