Multi-Modal Emotion Recognition Based on Wavelet Transform and BERT-RoBERTa: An Innovative Approach Combining Enhanced BiLSTM and Focus Loss Function

Abstract

Emotion recognition plays an increasingly important role in today’s society and has a high social value. However, current emotion recognition technology faces the problems of insufficient feature extraction and imbalanced samples when processing speech and text information, which limits the performance of existing models. To overcome these challenges, this paper proposes a multi-modal emotion recognition method based on speech and text. The model is divided into two channels. In the first channel, the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) feature set is extracted from OpenSmile, and the original eGeMAPS feature set is merged with the wavelet transformed eGeMAPS feature set. Then, speech features are extracted through a sparse autoencoder. The second channel extracts text features through the BERT-RoBERTa model. Then, deeper text features are extracted through a gated recurrent unit (GRU), and the deeper text features are fused with the text features. Emotions are identified by the attention layer, the dual-layer Bidirectional Long Short-Term Memory (BiLSTM) model, and the loss function, combined with cross-entropy loss and focus loss. Experiments show that, compared with the existing model, the WA and UA of this model are 73.95% and 74.27%, respectively, on the imbalanced IEMOCAP dataset, which is superior to other models. This research result effectively solves the problem of feature insufficiency and sample imbalance in traditional sentiment recognition methods, and provides a new way of thinking for sentiment analysis application.