Human-robot interaction for robotic manipulator programming in Mixed Reality

The paper presents an approach for interactive programming of the robotic manipulator using mixed reality. The developed system is based on the HoloLens glasses connected through Robotic Operation System to Unity engine and robotic manipulators. The system gives a possibility to recognize the real robot location by the point cloud analysis, to use virtual markers and menus for the task creation, to generate a trajectory for execution in the simulator or on the real manipulator. It also provides the possibility of scaling virtual and real worlds for more accurate planning. The proposed framework has been tested on pick-and-place and contact operations execution by UR10e and KUKA iiwa robots.