Deception detection using multimodal fusion approaches

Automatic deception detection is an important task that has gained a huge interest in different fields due to its potential applications. Particularly, it can improve justice and security in society by helping in detecting deceivers in high-stakes situations across jurisprudence, law enforcement, and national security domains, among others. However, the existing deception detection systems until today are not as accurate as it is expected, which makes their use very risky especially in critical fields. This article outlines an approach for automatically distinguishing between deceit and truth based on audio, video and text modalities and explores the possibility of combining them together in order to detect deception more accurately. First, each modality has been evaluated separately and then a feature and decision-level fusion approaches have been proposed to combine the considered modalities. The proposed feature level fusion approach investigates a diversified feature selection techniques to select the most relevant ones among the whole used feature set, while the decision level fusion approach is based on the belief theory considering information about the certainty degree of each modality. To do so, we used a real-life video dataset of people communicating truthfully or deceptively collected from public american court trials. Unimodal models trained on audio, video and text separately achieved an accuracy rate of 60%, 94% and 58% respectively. When using the feature level fusion approach, the best accuracy deception detection result reaches 93% using only 19 combined features, while a 100% deception recognition rate has been obtained with the decision-level fusion proposed approach, outperforming the results obtained in the literature.