Coordination of verbal and non-verbal actions in human―robot interaction at museums and exhibitions

Abstract In this article we analyze videotaped data in Japanese of naturally occurring human–human and experimental human–robot interaction in museums and exhibitions with the goal of developing a robot that can provide explanations in these settings. Based on the initial analysis of interaction among human (i) visitors and (ii) visitors and guides, we observe that verbal and non-verbal actions play an important role in directing others’ actions. In particular, in the visitor–guide interactions we discovered that guides make head turns from the exhibit towards a visitor and also point towards features of the exhibit, and that these non-verbal actions are consequential in gaining visitors’ response. Furthermore, guides turn their heads at particular places in their talk. We then programmed such coordination of verbal and non-verbal actions in a robot guide and conducted several experimental observations. These observations show that when the robot's talk was coordinated with non-verbal actions, visitors more often responded with their own non-verbal actions such as head nods. The results suggest that in developing robots to employ in interaction with humans as museum and exhibition guides, we need to consider the precise coordination of verbal and non-verbal actions.