2023 · 21 citations · 18 references
ComputingCommunicationInteraction DesignNatural Language ProcessingNonverbal CommunicationConversational InterfacesConversational User InterfacesDialogue SystemsEmotion RecognitionMultimodal AnalysisMultimodal InteractionContext RepresentationInteraction ModelingMulti-modal InteractionMultimodal Affective ComputingMultimodal ConversationMultimodal Emotion RecognitionMultimodal Machine LearningPath Selection
The multimodal emotion recognition in conversation task aims to predict the emotion label for a given utterance with its context and multiple modalities. Existing approaches achieve good results but also suffer from the following two limitations: 1) lacking modeling of diverse dependency ranges, i.e., long, short, and independent context-specific representations and without consideration of the different recognition difficulty for each utterance; 2) consistent treatment of the contribution for various modalities. To address the above challenges, we propose the Self-adaptive Context and Modal-interaction Modeling (SCMM) framework. We first design the context representation module, which consists of three submodules to model multiple contextual representations. Thereafter, we propose the modal-interaction module, including three interaction submodules to make full use of each modality. Finally, we come up with a self-adaptive path selection module to select an appropriate path in each module and integrate the features to obtain the final representation. Extensive experiments under four settings on three multimodal datasets, including IEMOCAP, MELD, and MOSEI, demonstrate that our proposed method outperforms the state-of-the-art approaches.
18
Densely Connected Convolutional Networks
Gao Huang, Zhuang Liu, Laurens van der Maaten et al. · 2017 · 44.7K citations
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
IEMOCAP: interactive emotional dyadic motion capture database
Carlos Busso, Murtaza Bulut, Chi-Chun Lee et al. · Language Resources and Evaluation · 2008 · 3.6K citations
Multimodal fusion for multimedia analysis: a survey
Pradeep K. Atrey, M. Anwar Hossain, Abdulmotaleb El Saddik et al. · Multimedia Systems · 2010 · 1.2K citations