2019 · 53 citations · 13 references
MedicineBehavioral SciencesNeuroscienceAffective NeuroscienceEye MovementsSensor FusionAffective ComputingAffective State DetectionElectroencephalographyEmotion ProcessingEmotion RecognitionEye Movement AnalysisClassifier FusionEmotionsMultimodal Affective ComputingMultimodal Affective State RecognitionMultimodal Emotion Recognition
Recently, various multimodal approaches to enhancing the performance of affective models have been developed. In this paper, we investigate the complementary representation properties of EEG and eye movement signals on classification for five human emotions: happy, sad, fear, disgust, and neutral. We compare the performance of single modality and two different modality fusion approaches. The results indicate that EEG is superior to eye movements in classifying happy, sad and disgust emotions, whereas eye movements outperform EEG in recognizing fear and neutral emotions. Compared with eye movements, EEG has the advantage of classifying the five emotions, with the mean accuracies of 69.50% and 59.81%, respectively. Due to the complementary representation properties, the modality fusion with bimodal deep auto-encoder significantly improves the classification accuracy to 79.71%. Furthermore, we study the neural patterns of five emotion states and the recognition performance of different eye movement features. The results reveal that five emotions have distinguishable neural patterns and pupil diameter has a relatively high discrimination ability than the other eye movement features.
13
The pupil as a measure of emotional arousal and autonomic activation
Margaret M. Bradley, Laura Miccoli, Miguel A. Escrig et al. · Psychophysiology · 2008 · 1.9K citations · Full text
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Wei‐Long Zheng, Wei Liu, Bao‐Liang Lu et al. · IEEE Transactions on Cybernetics · 2018 · 1.2K citations