Offset or Onset Frame: A Multi-Stream Convolutional Neural Network with CapsuleNet Module for Micro-expression Recognition

Micro-expression is a spontaneous facial expression, which may reveal people's real emotions. The micro-expression recognition has recently attracted much attention in psychology and computer vision community. In this paper, we designed a multi-stream Convolutional Neural Network (CNN) combined with the Capsule Network(CapsNet) module.named CNNCapsNet, to improve the performance of micro-expression recognition. Firstly, both vertical and horizontal optical flow are computed from the onset to the apex, and from the apex to the offset frame respectively, which is the first time that the offset frame information has been taken into account in the field of micro-expression recognition. Secondly, these four optical flow images and the grayscale image of apex frame are input into the five-stream CNN model to extract features. Finally, CapsNet completes micro-expression recognition by learning the features extracted by CNN. The method proposed in this paper are evaluated using the Leave-One-Subject-Out (LOSO) cross-validation protocol on CASME II. The results show that the offset information, which is often neglected, is more important than onset information for the recognition task. Our CNNCapsNet framework can achieve the accuracy of 64.63% for the five-class micro-expression classification.