Audio-visual and multimodal speech-based systems

Communication between humans uses many modalities. We communicate not only via verbal language, but also through our use of intonation, gaze, hand gestures, body gestures, and facial expressions. Using these modalities, we can add, modify, and substitute information in spoken conversations. Complementary use of several modalities in human-to-human communication ensures high accuracy, and only few communication problems occur. When communication problems do occur, conversation partners can easily recover, using the redundancy and complementarity of modalities. The goal of research on multimodal systems is to investigate how human—computer interaction can benefit from multiple modalities in similar ways.