Weakly Supervised Emotion Intensity Prediction for Recognition of Emotions in Images

Abstract

Recognition of emotions in images is attracting increasing research attention. Recent studies show that using local region information helps to improve the recognition performance. Intuitively, emotion intensity maps provide more detailed information than image regions. Inspired by this intuition, we propose an end-to-end deep neural network for image emotion recognition leveraging emotion intensity learning. The proposed network is composed of a first classification stream, an intensity prediction stream and a second classification stream. The intensity prediction stream is built on top of the feature pyramid network to extract multilevel features. The class activation mapping technique is used to generate pseudo intensity maps from the first classification stream to guide the proposed network for emotion intensity learning. The predicted intensity map is integrated into the second classification stream for final emotion recognition. The three streams are trained cooperatively to improve the performance. We evaluate the proposed network for both emotion recognition and sentiment classification on different benchmark datasets. The experimental results demonstrate that the proposed network achieves improved performance compared to previous state-of-the-art approaches.