Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using\n Affective Cues

Abstract

We present a learning-based method for detecting real and fake deepfake\nmultimedia content. To maximize information for learning, we extract and\nanalyze the similarity between the two audio and visual modalities from within\nthe same video. Additionally, we extract and compare affective cues\ncorresponding to perceived emotion from the two modalities within a video to\ninfer whether the input video is "real" or "fake". We propose a deep learning\nnetwork, inspired by the Siamese network architecture and the triplet loss. To\nvalidate our model, we report the AUC metric on two large-scale deepfake\ndetection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach\nwith several SOTA deepfake detection methods and report per-video AUC of 84.4%\non the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of\nour knowledge, ours is the first approach that simultaneously exploits audio\nand video modalities and also perceived emotions from the two modalities for\ndeepfake detection.\n