TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

TLDR

Multimodal sentiment analysis predicts speaker sentiment from textual, visual, and acoustic cues, but effective fusion of these modalities remains challenging, and few approaches use end‑to‑end translation models to capture subtle cross‑modal correlations. This study introduces TransModality, an end‑to‑end fusion method that applies Transformer‑based translation between modalities to improve multimodal sentiment analysis. TransModality treats modality translation as a means to generate joint representations, using a Transformer to encode features from both source and target modalities, and is evaluated on CMU‑MOSI, MELD, and IEMOCAP datasets. Experimental results demonstrate that TransModality achieves state‑of‑the‑art performance on these multimodal sentiment datasets.

Abstract

Multimodal sentiment analysis is an important research area that predicts speaker’s sentiment tendency through features extracted from textual, visual and acoustic modalities. The central challenge is the fusion method of the multimodal information. A variety of fusion methods have been proposed, but few of them adopt end-to-end translation models to mine the subtle correlation between modalities. Enlightened by recent success of Transformer in the area of machine translation, we propose a new fusion method, TransModality, to address the task of multimodal sentiment analysis. We assume that translation between modalities contributes to a better joint representation of speaker’s utterance. With Transformer, the learned features embody the information both from the source modality and the target modality. We validate our model on multiple multimodal datasets: CMU-MOSI, MELD, IEMOCAP. The experiments show that our proposed method achieves the state-of-the-art performance.