SGS2Net: deep representation of facial expression by graph-preserving sparse coding

Recently, deep learning has developed rapidly and made great improvements in facial expression recognition. However, deep learning has black box properties that lead to poor interpretability of the results. Differentiable programming provides a new aspect to balance the interpretability and convenience of deep learning. It provides a way to solve sparse coding problem that has solid mathematical foundations through recurrent neural network end-to-end. However, sparse representation is a traditional unsupervised learning method, and it does not effectively exploit the supervised information, which is helpful for facial expression recognition. We propose a differentiable programming algorithm that is called supervised graph-preserving sparse two neural network (SGS2Net) by exploiting both sparse and graph-preserving properties. The graph-preserving constraint may contain the class information. Therefore, the new model can be conducted as an unsupervised or a supervised way. A sparse representation of the facial images is obtained by minimizing the l1-norm of the coefficients, and the neighborhood of the samples is preserved by retaining the graph structure. The optimization procedure is conducted by gradient descent and threshold shrinkage and implemented by a new deep network structure end-to-end. SGS2Net is applied to facial micro-expression recognition and facial expression-based pain assessment, and it enhances the recognition accuracy by more than 10% and 4%, respectively, compared to state-of-the-art. It derives the fact that graph-preserving constraint improves the discriminant property of the network greatly, and SGS2Net has solid interpretability from sparse representation and graph embedding theory.