Non-Contact Emotion Recognition Combining Heart Rate and Facial Expression for Interactive Gaming Environments

A key to optimize a user’s entertainment or learning experience when playing interactive games is to understand his emotional responses. Current methods mostly exploit intrusive physiological signals to detect a player’s emotions. In this study, we proposed a method to detect a player’s emotions based on heart beat (HR) signals and facial expressions (FE). In this work, a continuous recognition of HR and FE through videos captured by Kinect2.0 is conducted considering the continuous perception of the human emotion. Bidirectional long and short term memory (Bi-LSTM) network is used to learn the HR features, and convolutional neural network (CNN) is trained to learn the FE features. To further meet the demands for real-time, the SOM-BP network is employed to fuse the HR and FE features, which can perfectly recognize the player’s emotion. Experimental results demonstrate our model has high accuracy and low computation time for four emotions of “excitement”, “anger”, “sadness” and “calmness” in different games. Moreover, the emotion’s intensity can be estimated by the HR value.