Facial Expression Based Imagination Index and a Transfer Learning Approach to Detect Deception

Abstract

In this paper, we introduce a framework to automatically distinguish between facial expression sequences associated with imagining vs. remembering while answering a question. Our experiment includes a baseline and relevant questioning technique in the context of deception with 220 participants (20 hours long). Baseline questioning includes participants being separately asked to remember and imagine an arbitrary experience. During the relevant questioning, participants were prompted to either lie or tell the truth about a certain task. We trained a neural network model on the baseline data and achieved an accuracy of 60% on classifying imagining vs. remembering, whereas human performance for this task is 51%. Relevant questioning included a set of questions, each of which became an independent response segment. Using a transfer learning approach, we use the pretrained model from the baseline to obtain an imagination probability score for each relevant response segment. We define this individual probability per response as the Imagination Index. We apply the imagination indices as a feature vector to classify the whole relevant section as truth vs. bluff with an accuracy of 70%, significantly outperforming the human performance of 52%.