The MuSe 2021 Multimodal Sentiment Analysis Challenge

TLDR

MuSe 2021 is a multimodal sentiment analysis challenge that integrates audio‑visual, language, and biological signals to evaluate sentiment, emotion, physiological‑emotion, and stress recognition. Its goal is to unite audio‑visual emotion, sentiment analysis, and health informatics communities. The challenge comprises four sub‑tasks—MuSe‑Wilder, MuSe‑Stress, MuSe‑Sent, and MuSe‑Physio—using the MuSe‑CaR review dataset and the Ulm‑TSST stress‑deposition dataset, with a baseline LSTM‑RNN and extracted feature sets. Baseline performance achieved CCCs of .4616 (MuSe‑Wilder), .5088 (MuSe‑Stress), .4908 (MuSe‑Physio) and an F1 of 32.82 % for MuSe‑Sent.

Abstract

Multimodal Sentiment Analysis (MuSe) 2021 is a challenge focusing on the tasks of sentiment and emotion, as well as physiological-emotion and emotion-based stress recognition through more comprehensively integrating the audio-visual, language, and biological signal modalities. The purpose of MuSe 2021 is to bring together communities from different disciplines; mainly, the audio-visual emotion recognition community (signal-based), the sentiment analysis community (symbol-based), and the health informatics community. We present four distinct sub-challenges: MuSe-Wilder and MuSe-Stress which focus on continuous emotion (valence and arousal) prediction; MuSe-Sent, in which participants recognise five classes each for valence and arousal; and MuSe-Physio, in which the novel aspect of 'physiological-emotion' is to be predicted. For this year's challenge, we utilise the MuSe-CaR dataset focusing on user-generated reviews and introduce the Ulm-TSST dataset, which displays people in stressful depositions. This paper also provides detail on the state-of-the-art feature sets extracted from these datasets for utilisation by our baseline model, a Long Short-Term Memory-Recurrent Neural Network. For each sub-challenge, a competitive baseline for participants is set; namely, on test, we report a Concordance Correlation Coefficient (CCC) of .4616 CCC for MuSe-Wilder; .5088 CCC for MuSe-Stress, and .4908 CCC for MuSe-Physio. For MuSe-Sent an F1 score of 32.82% is obtained.