2020 · 87 citations · 22 references
ComputingComputer ScienceComputer VisionMachine LearningNatural Language ProcessingAffective ComputingAttention NetworksDeep LearningEmotion RecognitionMultimodal InteractionSpeech RecognitionAttentionAttention-based FusionEmotion Recognition From SpeechHybrid Network ModelsMulti-modal InteractionMultimodal Emotion RecognitionMultimodal Feature FusionMultimodal Fusion
Emotion represents an essential aspect of human speech that is manifested in speech prosody.Speech, visual, and textual cues are complementary in human communication.In this paper, we study a hybrid fusion method, referred to as multi-modal attention network (MMAN) to make use of visual and textual cues in speech emotion recognition.We propose a novel multimodal attention mechanism, cLSTM-MMA, which facilitates the attention across three modalities and selectively fuse the information.cLSTM-MMA is fused with other uni-modal subnetworks in the late fusion.The experiments show that speech emotion recognition benefits significantly from visual and textual cues, and the proposed cLSTM-MMA alone is as competitive as other fusion methods in terms of accuracy, but with a much more compact network structure.The proposed hybrid network MMAN achieves state-of-the-art performance on IEMOCAP database for emotion recognition.
22
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
Large-Scale Video Classification with Convolutional Neural Networks
Andrej Karpathy, George Toderici, Sanketh Shetty et al. · 2014 · 6.3K citations
IEMOCAP: interactive emotional dyadic motion capture database
Carlos Busso, Murtaza Bulut, Chi-Chun Lee et al. · Language Resources and Evaluation · 2008 · 3.6K citations
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Zhun Liu, Ying Shen, Varun Lakshminarasimhan et al. · 2018 · 993 citations · Full text