Facial expression recognition with temporal modeling of shapes

TLDR

CRFs enable discriminative sequence labeling, while LDCRFs extend them with hidden states to capture motion sub‑structures and label dynamics. To develop an LDCRF‑based framework that automatically recognizes facial expressions from continuous video by modeling temporal shape variations. The framework uses LDCRFs to model temporal shape variations and applies PCA to assess class separability in reduced‑dimensional spaces. The LDCRF approach outperforms CRFs and SVMs, demonstrating that temporal shape variations are essential for accurate expression recognition, especially for subtle motions, whereas relying solely on appearance changes is insufficient.

Abstract

Conditional Random Fields (CRFs) can be used as a discriminative approach for simultaneous sequence segmentation and frame labeling. Latent-Dynamic Conditional Random Fields (LDCRFs) incorporates hidden state variables within CRFs which model sub-structure motion patterns and dynamics between labels. Motivated by the success of LDCRFs in gesture recognition, we propose a framework for automatic facial expression recognition from continuous video sequence by modeling temporal variations within shapes using LDCRFs. We show that the proposed approach outperforms CRFs for recognizing facial expressions. Using Principal Component Analysis (PCA) we study the separability of various expression classes in lower dimension projected spaces. By comparing the performance of CRFs and LDCRFs against that of Support Vector Machines (SVMs), we demonstrate that temporal variations within shapes are crucial in classifying expressions especially for those with a small range of facial motion like anger and sadness. We also show empirically that only using changes in facial appearance over time, without using shape variations, is not sufficient to obtain high performance for facial expression recognition.