IEEE/ACM Transactions on Audio Speech and Language Processing · 2023 · 13 citations · 39 references
ComputingComputer ScienceFeature EngineeringMachine LearningNatural Language ProcessingAffective ComputingContext-aware FusionEmotion RecognitionFeature-level FusionOpinion MiningSentiment AnalysisAdaptive FusionContext-adaptive FusionContextual Feature FusionMultimodal Emotion RecognitionMultimodal Feature FusionMultimodal Feature RepresentationMultimodal FusionSentiment Classification
Recently, video sentiment computing has become the focus of research because of its benefits in many applications such as digital marketing, education, healthcare, and so on. The difficulty of video sentiment prediction mainly lies in the regression accuracy of long-term sequences and how to integrate different modalities. In particular, different modalities may express different emotions. In order to maintain the continuity of long time-series sentiments and mitigate the multimodal conflicts, this paper proposes a novel Context-Based Adaptive Multimodal Fusion Network (CAMFNet) for consecutive frame-level sentiment prediction. A Context-based Transformer (CBT) module was specifically designed to embed clip features into continuous frame features, leveraging its capability to enhance the consistency of prediction results. Moreover, to resolve the multi-modal conflict between modalities, this paper proposed an Adaptive multimodal fusion (AMF) method based on the self-attention mechanism. It can dynamically determines the degree of shared semantics across modalities, enabling the model to flexibly adapt its fusion strategy. Through adaptive fusion of multimodal features, the AMF method effectively resolves potential conflicts arising from diverse modalities, ultimately enhancing the overall performance of the model. The proposed CAMFNet for consecutive frame-level sentiment prediction can ensure the continuity of long time-series sentiments. Extensive experiments illustrate the superiority of the proposed method especially in multimodal conflicts videos.
39
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al. · arXiv (Cornell University) · 2020 · 21.6K citations · Full text
An argument for basic emotions
Paul Ekman · Cognition & Emotion · 1992 · 9.4K citations
IEMOCAP: interactive emotional dyadic motion capture database
Carlos Busso, Murtaza Bulut, Chi-Chun Lee et al. · Language Resources and Evaluation · 2008 · 3.6K citations