Feature Extraction of Video Data for Automatic Visual Tool Tracking in Robot Assisted Surgery

The strong interest in data-driven surgical procedures has motivated the need for automatically analyzing instrument trajectories during surgical procedures to improve the efficacy of modern robot-assisted surgeries. We proposed a single view camera-based approach for automatic video analysis of surgical instruments during robot-assisted minimally invasive procedures. This video data-driven approach uses feature extraction for visual tracking of the surgical instrument, coupled with the kinematics of the pivoted motion in minimally invasive surgery for tool trajectory analysis. The self-contained nature of the approach enables the acquisition of trajectory information without any prior placement of markers or post storage of robot joint trajectories information. Based on visual inspection of the 2D localization results in the image frames, the proposed method demonstrated reasonable recovery of the 3D positional information of the surgical tools. Driven by the increasingly sophisticated interventional procedures of modern surgeries, this proposed trajectory tracking method provides a means for leveraging the wealth of surgical video data for data analytics in Surgical Data Science. The long-term goal of the study is to contribute towards the advancement of Surgical Data Science through the establishment of a video-based data acquisition technique.