A smart operator advice model by deep learning for motion recognition in human–robot coexisting assembly line

Operator advice and guidance systems can help junior workers follow standard operating procedures, which are critical to a human–robot coexisting assembly line to guarantee harmonious and complete tasks with high quality. In this study, a smart operator advice model by deep learning is proposed. Two mechanisms are built as the model core in the model, which is an object detection mechanism using convolutional neural network and a motion recognition mechanism using a decision tree classifier. Object detection is carried out by three independent cameras monitoring all of the objects in the system. The study implements the proposed model in a graphic processing unit (GPU) card assembly line consisting of objects such as operator, robot, screwing machine, fan, motherboard, GPU card, screwdriver, hand, and body. The objects are identified by the object detection mechanism by three parallel and independent cameras. Motion detection is achieved by three parallel and independent decision tree classifiers for motion recognition in the three cameras where the inputs are the object coordinates and speeds. Through a majority rule, the right motion with the highest votes is confirmed. The final output is the task checking and advice provided by the proposed operator advice model. Results are evaluated through the GPU final assembly line. F1-score of 0.966 shows a promising performance. This smart model facilitates informative instruction to the junior operator when conducting complex assembly activities in real time.