Facial feature tracking and pose estimation in video sequences by factorial coding of the low-dimensional entropy manifolds due to the partial symmetries of faces

Abstract

In a number of practical scenarios, such as video conferencing and visual human/computer interaction, objects that belong to a well defined class are segmented, normalized, and encoded, after which they are stored and/or transmitted, and subsequently reconstructed. The Karhunen-Loeve transform (KLT) optimally concentrates the signal power in a relatively small number of uncorrelated coefficients. Nevertheless, it implicitly assumes a multidimensional Gaussian probability model, which is typically not correct. Here we show that, in the context of video sequences of human heads, the segmentation and normalization steps result in partial symmetries which force the KLT coefficients to lie close to low-dimensional manifolds in suitably chosen high-dimensional KLT subspaces. We show how this fact can be used to track the faces robustly, and to estimate their pose. We use vector quantization to discover those manifolds, and to build a factorial code that has a substantially lower dimensionality than KLT.