Simulation of Robot-Assisted Flexible Needle Insertion Using Deep Q-Network

Flexible needle insertion with bevel tips is becoming a preferred method for approaching targets in the human body in the least invasive manner. However, to successfully implement needle insertion, surgeons require prolonged training processes and long-term experience to develop essential handling skills. This paper presents a new path planning approach with Deep Reinforcement Learning (DRL) to implement automatic needle insertion using a surgical robot. In this paper, Deep Q-Network (DQN) algorithm is utilized to learn the control policy for flexible needle steering with needle-tissue interaction. As the human body is composed of a complex environment such as tissues, blood vessels, bones, and muscles, the uncertainty of the needle-tissue interaction should be considered during insertion. To model this complex interaction in path planning, utilizing a neural network to approximate the action-value function is more efficient than using traditional array methods in terms of time and accuracy. In our simulation, the agent (needle) can be controlled with 2 degrees of freedom (bevel direction rotation and insertion) and received negative rewards when it collides with obstacles, goes out of range, or exceeds a predefined number of rotations. During the training, the agent demonstrates the accuracy and efficiency of the learned policy through feedback scores in every episode. In addition, this system incorporates the uncertainty within flexible needle-tissue interaction using a stochastic environment. Compared with other traditional methods for flexible needle path planning, we demonstrated that motion planning of bevel-tip flexible needles in complex human bodies using DRL has better efficiency and accuracy.