Voice Dialogue with a Collaborative Robot Driven by Multimodal Semantics

The paper describes the control system of an autonomous mobile service robot using a human-machine dialogue to control the robot in a way natural for human with the help of syntactic structures that define the sequence of certain actions. The interaction with the robot by means of natural language in the formation of target designations is provided by the multimodal representation of the working area in the form of a hybrid map of the surrounding space with highlited work areas and locations of key objects complemented by a set of semantic relations that characterize features of the robot operation and impact of the environmental changes on its operation. The article discusses the control system of the mobile service robo, performing transport and logistics operations in a public place in direct contact with lots of people around. The proposed approach to the dialog control of the robot based on the multimodal representation of the robot operation area not only simplifies the process of determining the sequence of operations, but also enhances the level of safety for other people, objects and the robot itself in the process of fulfilling its service tasks.