2020 · 167 citations · 17 references
ComputingComputer ScienceComputer VisionMachine LearningSensor FusionAffective ComputingAttention NetworksDeep LearningEmotion RecognitionFeature-level FusionMultimodal InteractionAudio-visual SystemsFeature FusionFusion StrategiesModel FusionMultimodal Emotion RecognitionMultimodal FusionRepresentation FusionTransformer Models
Multimodal fusion increases the performance of emotion recognition because of the complementarity of different modalities. Compared with decision level and feature level fusion, model level fusion makes better use of the advantages of deep neural networks. In this work, we utilize the Transformer model to fuse audio-visual modalities on the model level. Specifically, the multi-head attention produces multimodal emotional intermediate representations from common semantic feature space after encoding audio and visual modalities. Meanwhile, it also can learn long-term temporal dependencies with self-attention mechanism effectively. The experiments, on the AVEC 2017 database, shows the superiority of model level fusion than other fusion strategies. Moreover, we combine the Transformer model and LSTM to further improve the performance, which achieves better results than other methods.
17
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · 2023 · 76.3K citations · Full text
A Concordance Correlation Coefficient to Evaluate Reproducibility
Biometrics · 1989 · 8.7K citations
Florian Eyben, Martin Wöllmer, Björn W. Schuller · 2010 · 2.6K citations · Full text
Analysis of emotion recognition using facial expressions, speech and multimodal information
Carlos Busso, Zhigang Deng, Murtaza Bulut et al. · 2004 · 832 citations