Cultural differences in pattern matching: multisensory recognition of socio-affective prosody

This study focuses on the cross-cultural differences in perception of audio visual prosodic recordings of Japanese social affects, which have been largely uninvestigated. The study compares cultural differences of perceptual patterns of 21 Japanese subjects with 20 French subjects who have no knowledge of Japanese language or Japanese social affects. The test material was a semantically affectively neutral utterance expressed in 9 various social affects by 2 Japanese speakers (one male, one female) who were recognized as best performers in our previous recognition experiment. The perception test task was to first choose the speaker’s intended social affective meaning from the audio alone and also the video alone presentations, using a forced choice paradigm. The next task was to match the audio (no video) with the video (no audio) information. The results revealed that the native subjects could correctly combine auditory and visually expressed social affects, showing some confusion inside semantic categories. Different matching patterns were seen for non-native subjects.