Context sensitivity in the detection of changes in facial emotion

Facial expressions that convey emotional information in communication is an important topic in the scientific study of emotion. In particular, extensive research in the area of emotion has been conducted on the recognition of facial expressions; that is, the identification and discrimination between emotions using facial expressions as cues1,2,3,4. Additionally, during actual communication, the meaning of facial expressions must be determined by considering the situational background/context. Indeed, the relationship between the recognition of facial expressions and its background/context has been investigated extensively from diverse perspectives, using behavioural, physiological, and neuroimaging techniques. For example, the identification of emotions associated with facial expressions has been reported to be significantly affected by contextual inputs, which include body postures5,6,7, voices8,9, background scenes10,11, and words12,13. Moreover, the influence of context is inevitable and persists even when subjects are instructed to ignore such information7. However, because these studies focused on the integration of multi-modal sensory stimuli, the temporal aspect of context, i.e. the relationship between contextual information and the resulting changes in others’ emotional states, was considered as being outside their scope. The temporal aspect of context in the recognition of emotion has been investigated extensively using a well-established experimental paradigm of mood-congruent/affective priming effects14,15,16,17. Affective priming is a phenomenon in that the evaluation of a positive or negative target preceded by a valence-congruent prime (i.e. positive prime–positive target or a negative prime–negative target) are faster and more accurate compared to an incongruent prime (i.e., positive prime–negative target or negative prime–positive target)14,15,16,17. According to the prevailing theory, affective priming effects emerge because the affective prime activates the memory representation of the affectively-related target, resulting in a faster and more accurate response to the target with the same emotional valence14,15,17. Although such processes may play a crucial role in the integration of temporal context in emotion recognition, in everyday life, matching between a preceding stimulus and a following stimulus is not enough for the integration of temporal context in emotion recognition. For example, an individual might show a facial expression indicating extreme happiness upon receiving a gift, but after opening the wrapping, this expression might shift to indicate slight happiness. Even though the individual’s facial expression still can be identified as a positive emotion, the transition from very happy to slightly happy might suggest a negative shift in emotion: he/she might be a little disappointed by the gift. In such situations, rather than a valence, the direction of change in facial expression associated with a context has a crucial meaning. Indeed, recent studies have emphasised the importance of the change in direction of facial expression in emotion recognition18,19. These studies found that video-clips of dynamically changing facial expressions can induce the anticipatory process of facial expression change, and this anticipatory process might be involved in mindreading mechanisms, such as theory of mind19. In addition to the ability to read ‘changes’ in others’ facial expressions, the meanings of changes in facial expression must be determined by integrating the situational background/context. Moreover, the critical role of background/context information in social communication implies that impaired recognition of emotion, resulting from poor integration of contextual information might be related to mental disorders, including autism spectrum disorders (ASD) and alexithymia, which involve impairments in describing and identifying emotions20,21. However, an experimental paradigm to investigate the mechanism underlying the integration of contextual information in the recognition of changes in facial expressions has not yet been fully explored. In order to address this issue, the current study aimed to develop a novel behavioural task to measure the effect of contextual information on the recognition of changes in facial expression. Additionally, we investigated the relationship between the ability to identify/describe emotions and individual differences in emotion recognition with contextual information.