Video-Based Facial Expression Recognition using Deep Temporal–Spatial Networks

Abstract

It’s a challenging task to recognize facial expression in video sequences due to the gap between the hand-crafted features and the subjective emotions. To bridge the gap, this paper proposes a novel method of video-based facial expression recognition using deep temporal–spatial networks. The proposed method firstly employs multimodal deep convolutional neural networks (CNN), including the spatial CNN network and the temporal CNN network, to extract high-level spatial and temporal features in video sequences, respectively. The temporal–spatial CNN networks are fine-tuned on target video facial expression data from a pre-trained CNN model. Specially, the spatial network is used to learn deep spatial features from the static expression images in a video. Likewise, the temporal network is adopted to learn deep temporal features from the produced optical flow images between multiple frames in a video. Then the extracted spatial and temporal features are combined in a fusion network to conduct video-based facial expression classification tasks. Extensive experiments on two public video-based facial expression datasets, i.e. the BAUM-1s and RML database, demonstrate the promising performance of the proposed method, outperforming the-state-of-the-arts.