Analysis of Mood Changes and Facial Expressions during Cognitive Behavior Therapy through a Virtual Agent

In cognitive behavior therapy (CBT) with a virtual agent, facial expression processing is expected to be useful for dialogue response selection empathic dialogue. Unfortunately, its use in current works remains limited. One reason for this situation is the lack of research on the relationship between mood changes facial expressions through CBT-oriented interaction. This study confirms the improvement of negative moods through interaction with a virtual agent and identifying facial expressions that correlate with mood changes. Based on the cognitive restructuring of CBT, we created a fixed dialogue scenario and implemented it in a virtual agent. We recorded facial expressions during dialogues with 23 undergraduate and graduate students, calculated 17 types of action units (AUs), which are the units of facial movements, and performed a correlation analysis using the change rate of mood scores and the amount of the changes in the AUs. The mean mood improvement rate was 35%, and the mood improvements showed correlations with AU5 (r = -0.51), AU17 (r = 0.45), AU25 (r = -0.43), and AU45 (r = 0.45). These results imply that mood changes are reflected in facial expressions. The AUs identified in this study have the potential to be used for agent-interaction modeling.