Facial Emotion Recognition in UX Evaluation: A Systematic Review

In UX evaluation, pragmatic criteria still prevail over hedonic ones. However, emotion is an essential part of the user experience and has particular importance to system acceptance, thus it should be assigned more value in such assessments. Emotion recognition based on facial expressions is one of the tools that can be used to assess user emotion and it can be performed during the interaction and in a less intrusive way than with other sensors. In this context, this paper presents a systematic literature review about user experience evaluation using facial emotion recognition addressing the following research questions: which kinds of user studies take advantage of facial emotion recognition and to what purpose; how emotion recognition is implemented; how user experience is evaluated using this data and what strategies are used to validate these results. From 372 unique papers identified by the search string, 332 were initially discarded and, of the remaining 40 remaining papers that were read in full, only 14 were included in the final analysis. We identified that this area is still relatively novel, with few works published and all of them in the last eight years. Facial images were the most frequent type of data used and comparisons with self-reported emotions were the prevalent strategy to validate automatic emotion recognition, but just as often no such strategy was discussed.