An Automated Structural Approach to Support Theatrical Performances by Introducing Gesture Recognition to a Cuing System

In this article, we explain how to process and interpret stage directions in order to support actors’ performances. To this end, we specify the function of coverbal gesture and movement as an additional statement by an actor. We introduce a classification of communicative gestures by individuals and groups based on a description with typical body triangles and their sensor-detectable vertices moving in time. We claim that with the help of a speech and gesture recognition system, the occurrence of semantic isotopies to characterize a scene with speech and accompanying metaphoric gestures described in the stage directions can be detected and encoded in a feature vector. As a proof of concept, we highlight the ritual theater play Orphee d’Afrique, which was written as a second part of a homonymous chant novel and presents an implementation of a gesture- and voice-detection cuing system.