Multimodal Depression Detection Using Task-oriented Transformer-based Embedding

Abstract

Automatic depression detection from multimodal signals assumes significance since the features from different modalities possesses contrasting but relevant information. Multimodal signal analysis involving the derivation of a joint representation, identification of correct alignment, and selection of appropriate fusing mechanisms is a challenging task. In this paper, we focus on an approach to detecting depression from multimodal signals using task-specific representation and late fusion. Our method uses a task-oriented embedding generated through a GPT2-medium language model that is fine-tuned on a conversational dataset. We present our experimental results benchmarking on Distress Analysis Interview Corpus - a Wizard of Oz (DAIC-WOZ) corpus. Our proposed task-oriented embedding on a simple BiLSTM network outperforms the previous text-based depression prediction methods. The results are further strengthened when audio and video modalities are fused with the text predictions on a late-fusion network.