Modeling Natural Language Controlled Robotic Operations

There are multiple ways to control a robotic system. Most of them require the users to have prior knowledge about robots. Natural language based control is a very promising method due to its versatility, ease of use, and without the need for extensive training for novice users. Since natural language instructions from users cannot be understood by the robots directly, the linguistic input has to be processed into a formal representation which captures the task specification and removes the ambiguity inherent in natural language. For most of existing natural language controlled robotic system, they assume the given language instructions are already in correct orders. However, it is very likely for untrained users to give commands in a mixed order based on their direct observation and intuitive thinking. Simply following the order of the commands can lead to failures of tasks. To provide a remedy for the problem, we propose a framework named Dependency Relation Matrix (DRM) to model and organize the linguistic input, in order to figure out a feasible subtask sequence for later execution. Besides, the proposed approach projects both linguistic input and sensory information into the same space, and use the difference between the goal specification and temporal sensory feedback to drive the system moving forward. It also helps to monitor the progress of task execution by comparing the temporal sensory feedback and the goal configuration. In this paper, we describe the DRM framework in detail, and illustrate the utility of this approach with experiment results.