Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior

TLDR

The study systematically compares machine learning methods for fully automatic facial expression recognition and presents preliminary results on spontaneous expressions. The system was built using AdaBoost, SVM, and LDA classifiers, with AdaBoost for feature selection on Gabor filters, and was applied to FACS action unit recognition. Using AdaBoost‑selected Gabor filters and SVM, the system achieved 93 % accuracy on a 7‑way Cohn‑Kanade test, 94.8 % accuracy on 17 FACS action units, operates in real time, and its smooth classifier outputs enable measurement of expression.

Abstract

We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions. We report results on a series of experiments comparing recognition engines, including AdaBoost, support vector machines, linear discriminant analysis. We also explored feature selection techniques, including the use of AdaBoost for feature selection prior to classification by SVM or LDA. Best results were obtained by selecting a subset of Gabor filters using AdaBoost followed by classification with support vector machines. The system operates in real-time, and obtained 93% correct generalization to novel subjects for a 7-way forced choice on the Cohn-Kanade expression dataset. The outputs of the classifiers change smoothly as a function of time and thus can be used to measure facial expression dynamics. We applied the system to to fully automated recognition of facial actions (FACS). The present system classifies 17 action units, whether they occur singly or in combination with other actions, with a mean accuracy of 94.8%. We present preliminary results for applying this system to spontaneous facial expressions.