Computer Animation and Virtual Worlds · 2022 · 19 citations · 20 references
ComputingComputer ScienceFeature EngineeringNatural Language ProcessingVisual AttentionMultimodal InteractionSentiment AnalysisContrastive LearningMultimodal Feature FusionMultimodal Feature RepresentationMultimodal FusionMultimodal Representation LearningRedundant FeaturesSemantic FeaturesSparse Learning
Abstract Learning unimodal representations and improving multimodal fusion are two cores of multimodal sentiment analysis (MSA). However, previous methods ignore the information differences between different modalities: Text modality has high‐order semantic features than other modalities. In this article, we propose a sparse‐ and cross‐attention (SCANET) framework which has asymmetric architecture to improve performance of multimodal representation and fusion. Specifically, in the unimodal representation stage, we use sparse attention to improve the representation efficiency of two modalities and reduce the low‐order redundant features of audio and visual modalities. In the multimodal fusion stage, we design an innovative asymmetric fusion module, which utilizes audio and visual modality information matrix as weights to strengthen the target text modality. We also introduce contrastive learning to effectively enhance complementary features between modalities. We apply SCANET on the CMU‐MOSI and CMU‐MOSEI datasets, and experimental results show that our proposed method achieves state‐of‐the‐art performance.
20
Advances in neural information processing systems 7
Computers & Mathematics with Applications · 1996 · 14.4K citations
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
Zihang Dai, Zhilin Yang, Yiming Yang et al. · 2019 · 3.1K citations · Full text