Complementing Speech Interaction Design with Touch for Multi-Robot Systems

There has been increasing demand in using multiple robots with human intervention for higher robustness in sectors like agriculture, clinical applications, environment surveillance, military operations, security. The benefits of using multiple robots include increased speed of missions largely due to ability to do parallel tasks and redundancy due to being able to replace a robot with another. However, at the same time, multiple robots substantially increase amount of information exchange with their operator. Therefore, designing interactions between human and multiple robots to achieve their effective cooperation has been a difficult research issue. In this paper, we propose an intuitive complementary user-interface design using speech and touch to control multiple robots simultaneously for a mission. This is achieved using a list of touch complement speech guidelines. We tested the system with six subjects who operated one, two and three robots in a simulation environment built using Unity. The mean performance score for various tasks and perceived cognitive load for each experiment iteration measured using the NASA-TLX questionnaire is then shown to illustrate the benefits of touch complement speech.