Human activity recognition for efficient human-robot collaboration

A crucial technology in modern smart manufacturing is the human-robot collaboration (HRC) concept. In the HRC, operators, and robots unite and collaborate to perform complex tasks in a variety of scenarios, heterogeneous and dynamic conditions. A unique role in the implementation of the HRC model, as a means of sensation, is assigned to machine vision systems. It provides the receipt and processing of visual information about the environment, the analysis of images of the working area, the transfer of this information to the control system, and decision-making within the framework of the task. Thus, the task of recognizing the actions of a human-operator for the development of a robot control system in order to implement an effective HRC system becomes relevant. The operator commands fed to the robot can have a variety of forms: from simple and concrete to quite abstract. This introduces several difficulties when the implementation of automated recognition systems in real conditions; this is a heterogeneous background, an uncontrolled work environment, irregular lighting, etc. In the article, we present an algorithm for constructing a video descriptor and solve the problem of classifying a set of actions into predefined classes. The proposed algorithm is based on capturing three-dimensional sub-volumes located inside a video sequence patch and calculating the difference in intensities between these sub-volumes. Video patches and central coordinates of sub-volumes are built on the principle of VLBP. Such a representation of three-dimensional blocks (patches) of a video sequence by capturing sub-volumes, inside each patch, in several scales and orientations, leads to an informative description of the scene and the actions taking place in it. Experimental results showed the effectiveness of the proposed algorithm on known data sets.