Vision-guided fine-operation of robot and its application in eight-puzzle game

Industrial robots can perform delicate operations, and they have good stability and durability. However, robots adapt with difficulty to changes in tasks and environments, and they basically can only perform operations in a fixed logic sequence. In contrast, humans can adapt to changes in the environment at any time due to their strong hand–eye coordination. Humans can quickly adjust their limbs to adapt to changes in targets, distances, and directions, as observed by their visual system; this produces a perfect closed-loop control process called the sensor–actor process. This paper studies the robotic hand–eye coordination problem using a robot playing an eight-puzzle game. First, the robot’s system analyzes changes in the position, angle, and layout of the puzzle board in the scene through an image recognition algorithm. It then formulates a more optimized operation sequence. Next, the system transforms the instructions of moving tiles into the physical coordinate values in the world coordinate system according to the image-understanding result. The robot then moves its hand to accurately point at tiles and move them in the correct direction. These tasks require the robot to perceive various changes in the position, posture, and initial layout of the puzzle board in the scene, and to calculate the motion vector parameters. To validate the proposed approach, field experiments are conducted. The success rate of the move operation was over 96%, which shows that this system based on visual perception can greatly improve the adaptability of the robot, making it more flexible and autonomous, and laying the foundation for expanding the robot’s ability to work in spontaneous scenes.