Multi-modal emotion recognition using recurrence plots and transfer learning on physiological signals

In this paper we propose to use Recurrence Plots (RP) to generate 2D representations of physiological activity which should be less subject dependent and better suited for non-stationary signals such as EDA. The performance of spectrograms and RPs are compared on two publicly available datasets: AMIGOS and DEAP. Transfer learning is employed by using a pre-trained ResNet-50 model to recognize emotional states (high vs low arousal and high vs low valence) from the two types of representations. Results show that RPs reach a similar performance to spectrograms on periodic signals such as ECG and plethysmography (F1 of 0.76 for valence and 0.74 for arousal on the AMIGOS dataset) while they outperform spectrograms on EDA (F1 of 0.74 for valence and 0.75 for arousal). By combining the two sources of information we were able to reach a F1 of 0.76 for valence and 0.75 for arousal.