2024 · 16 citations · 14 references
ComputingData ScienceNatural Language ProcessingNonverbal CommunicationPsycholinguisticsDeep LearningEmotion RecognitionMultimodal AnalysisMultimodal CommunicationMultimodal InteractionSentiment AnalysisAttentionEmotionsMultimodal Emotion RecognitionMultimodal Feature RepresentationMultimodal Interaction AnalysisMultimodal Machine Learning
Multimodal sentiment analysis aims to utilize different modalities including language, visual, and audio to identify human emotions in videos. Multimodal interaciton mechanism is the key challenge. Previous works lack modeling of multimodal interaction at different grain levels, and does not suppress redundant information in multimodal interaction. This leads to incomplete multimodal representation with noisy information. To address these issues, we propose Multi-grained Multimodal Interaction Network (MMIN) to provide a more complete view of multimodal representation. Coarse-grained Interaction Network (CIN) exploits the unique characteristics of different modalities at a coarse-grained level and adversarial learning is used to reduce redundancy. Fine-grained Interaction Network (FIN) employ sparse-attention mechanism to capture fine-grained interactions between multimodal sequences across distinct time steps and reduce irrelevant fine-grained multimodal interaction. Experimental results on two public datasets demonstrate the effectiveness of our model in multimodal sentiment analysis.
14
Proceedings of the 23rd international conference on Machine learning
William W. Cohen, Andrew Moore · 2006 · 2.6K citations
Devamanyu Hazarika, Roger Zimmermann, Soujanya Poria · 2020 · 885 citations · Full text
Wenmeng Yu, Hua Xu, Ziqi Yuan et al. · Proceedings of the AAAI Conference on Artificial Intelligence · 2021 · 667 citations · Full text