Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

TLDR

Online videos are widely shared, and automatic analysis of such opinion videos presents new challenges in computational linguistics and multimodal analysis. The authors aim to exploit the dynamics between visual gestures and verbal messages to improve sentiment modeling. They introduce a new multimodal dataset with sentiment‑intensity annotations, analyze interaction patterns between facial gestures and spoken words, propose a multimodal dictionary representation, and evaluate it in a speaker‑independent sentiment‑intensity prediction setting. The study identifies four interaction types—neutral, emphasizer, positive, and negative—and shows that the multimodal dictionary representation significantly outperforms conventional early‑fusion approaches.

Abstract

People share their opinions, stories, and reviews through online video sharing websites every day. The automatic analysis of these online opinion videos is bringing new or understudied research challenges to the field of computational linguistics and multimodal analysis. Among these challenges is the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment. This article addresses this question in four ways: introducing the first multimodal dataset with opinion-level sentiment intensity annotations; studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity; proposing a new computational representation, called multimodal dictionary, based on a language-gesture study; and evaluating the authors' proposed approach in a speaker-independent paradigm for sentiment intensity prediction. The authors' study identifies four interaction types between facial gestures and verbal content: neutral, emphasizer, positive, and negative interactions. Experiments show statistically significant improvement when using multimodal dictionary representation over the conventional early fusion representation (that is, feature concatenation).