Correlated Attention Networks for Multimodal Emotion Recognition

Abstract

Emotion is a subjective, conscious experience when people face different kinds of stimuli. In this paper, we propose a new model, Correlated Attention Network (CAN), to make multimodal emotion recognition. Correlated Attention Network is an extension of attention based recurrent neural network with correlation calculated of different gated recurrent units to take correlation of EEG and eye movement extracted signals into attention mechanism and takes advantage of coordinated representation with complementary features. In experiments on 3 real world datasets, we find that our model can significantly contribute to higher emotion classification accuracy when higher correlation is acquired. Our experiment results indicate that the Correlated Attention Network model performs better than the state-of-the-art methods with a mean accuracy of of 94.03% on SEED dataset, 87.71% on SEED IV dataset, 88.51% and 85.62% for four classification and two dichotomies on DEAP dataset, respectively.