Classification of Individual’s discrete emotions reflected in facial microexpressions using electroencephalogram and facial electromyogram

Abstract Facial microexpressions are defined as brief, subtle, and involuntary movements of facial muscles reflecting genuine emotions that a person tries to conceal. Because microexpressions are involuntary and uncontrollable, automatic detection of microexpressions and recognition of emotions reflected in the microexpressions can be used in various applications. With the advancement of artificial-intelligence-based non-face-to-face interviews and computer-assisted treatment of mood disorders, the need for developing a technique to precisely detect microexpressions is gradually increasing. In this study, we developed facial electromyography (fEMG)- and electroencephalography (EEG)-based methods for the detection of microexpressions and recognition of emotions reflected in microexpressions as a potential alternative to computer vision-based methods. We first assessed the performance of microexpression detection, and then evaluated the performance of classification of the emotions reflected in the microexpressions. In our experiments with 16 participants, six discrete emotions could be classified using support vector machine with the best F1 score of 0.971 when optimal fEMG and EEG channels were selected, demonstrating the potential usability of the fEMG- and EEG-based emotion recognition method in practical scenarios. It is noteworthy that EEG was more useful for classifying discrete emotions compared to fEMG (best F1 scores: EEG–0.962; fEMG–0.797). To the best of our knowledge, this is the first study to estimate emotions reflected in facial microexpressions using EEG.