IEEE Transactions on Affective Computing · 2024 · 22 citations · 54 references
ComputingComputer ScienceComputer VisionFeature EngineeringNatural Language ProcessingAffective ComputingEmotion RecognitionMultimodal AnalysisOpinion MiningSentiment AnalysisVisual SemanticsAdaptive FusionCross-modal Feature AlignmentCross-modal FusionModal FusionMultimodal Affective ComputingMultimodal Feature FusionMultimodal FusionSentiment Classification
Recently, multi-modal affective computing has demonstrated that introducing multi-modal information can enhance performance. However, multi-modal research faces significant challenges due to its high requirements regarding data acquisition, modal integrity, and feature alignment. The widespread use of multi-modal pre-training methods offers the possibility of aiding visual sentiment analysis by introducing cross-domain knowledge. This paper proposes a Multi-Aspect Semantic Auxiliary Network (MASANet) for visual sentiment analysis. Specifically, MASANet achieves modality expansion through cross-modal generation, making it possible to introduce cross-domain semantic assistance. Then, a cross-modal gating module and an adaptive modal fusion module are proposed for aspect-level and cross-modal interaction, respectively. In addition, a designed semantic polarity constraint loss is presented to improve sentiment multi-classification performance. Evaluations of eight widely-used affective image datasets demonstrate that our proposed method outperforms the state-of-the-art methods. Further ablation experiments and visualization results also confirm the effectiveness of the proposed method and its modules.
54
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Show and tell: A neural image caption generator
Oriol Vinyals, Alexander Toshev, Samy Bengio et al. · 2015 · 6.4K citations
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Piyush Sharma, Nan Ding, Sebastian Goodman et al. · 2018 · 1.8K citations · Full text
Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning
Jiasen Lu, Caiming Xiong, Devi Parikh et al. · 2017 · 1.7K citations