IEEE Transactions on Affective Computing · 2024 · 24 citations · 34 references
ComputingLinguisticsNatural Language ProcessingNonverbal CommunicationPsycholinguisticsAffective ComputingDeep LearningEmotion RecognitionEmotion UnderstandingMultimodal InteractionSemantic NetworksCross-modal AlignmentCross-modal Feature AlignmentCross-modal FusionGraph RefinementJoint Space AnalysisMultimodal Emotion RecognitionSemantic ConsistencyUnification
Emotion recognition in conversation (ERC) based on multiple modalities has attracted enormous attention. However, most research simply concatenated multimodal representations, generally neglecting the impact of cross-modal correspondences and uncertain factors, and leading to the cross-modal misalignment problems. Furthermore, recent methods only considered simple contextual features, commonly ignoring semantic clues and resulting in an insufficient capture of the semantic consistency. To address these limitations, we propose a novel multi-level alignment and cross-modal unified semantic graph refinement network (MA-CMU-SGRNet) for ERC task. Specifically, a multi-level alignment (MA) is first designed to bridge the gap between acoustic and lexical modalities, which can effectively contrast both the instance-level and prototype-level relationships, separating the multimodal features in the latent space. Second, a cross-modal uncertainty-aware unification (CMU) is adopted to generate a unified representation in joint space considering the ambiguity of emotion. Finally, a dual-encoding semantic graph refinement network (SGRNet) is investigated, which includes a syntactic encoder to aggregate information from near neighbors and a semantic encoder to focus on useful semantically close neighbors. Extensive experiments on three multimodal public datasets show the effectiveness of our proposed method compared with the state-of-the-art methods, indicating its potential application in conversational emotion recognition. Implementation codes can be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zxiaohen/MA-CMU-SGRNet</uri> .
34
Momentum Contrast for Unsupervised Visual Representation Learning
Kaiming He, Haoqi Fan, Yuxin Wu et al. · 2020 · 12K citations
IEMOCAP: interactive emotional dyadic motion capture database
Carlos Busso, Murtaza Bulut, Chi-Chun Lee et al. · Language Resources and Evaluation · 2008 · 3.6K citations
Context-Dependent Sentiment Analysis in User-Generated Videos
Soujanya Poria, Erik Cambria, Devamanyu Hazarika et al. · 2017 · 875 citations · Full text
DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika et al. · 2019 · 813 citations