Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition

Abstract

A key challenge of facial expression recognition (FER) is to develop effective representations to balance the complex distribution of intra- and inter- class variations. The latest deep convolutional networks proposed for FER are trained by penalizing the misclassification of images via the softmax loss. In this paper, we show that better FER performance can be achieved by combining the deep metric loss and softmax loss in a unified two fully connected layer branches framework via joint optimization. A generalized adaptive (N+M)-tuplet clusters loss function together with the identity-aware hard-negative mining and online positive mining scheme are proposed for identity-invariant FER. It reduces the computational burden of deep metric learning, and alleviates the difficulty of threshold validation and anchor selection. Extensive evaluations demonstrate that our method outperforms many state-of-art approaches on the posed as well as spontaneous facial expression databases.