Multimodal Sentiment Recognition With Multi-Task Learning

Abstract

Sentiment recognition in social network aims at recognizing the underlying affective states of user-generated content. The research center of sentiment recognition is moving from pure texts to multimodal contents, with the explosive growth of social platforms. Different from the image-text contents on blog and review platforms, multimodal sequences play a dominant role in streaming media, e.g. YouTube and TikTok. Sentiment recognition for multimodal sequences needs to extract the common and specific information of modalities. But current studies only focus on learning the cross-modal fusion representation to explore the inter-modal interaction, while neglecting the interactions and characteristics within each modality. We propose a novel cascade and specific scoring model, which aims at learning better cross-modal and unimodal representations to capture both the inter- and intra-modal interactions for sentiment recognition. Qualitative and quantitative experiments on two benchmarks have demonstrated the competitive performances of the proposed methods.