FlexiVision: Teleporting the Surgeon’s Eyes via Robotic Flexible Endoscope and Head-Mounted Display

A flexible endoscope introduces more dexterity to the image capturing in endoscopic surgery. However, manual control or automatic control based on instrument tracking does not handle the misorientation between the endoscopic video and the surgeon. We propose an automatic flexible endoscope control method that tracks the surgeon’s head with respect to the object in the surgical scene. The robotic flexible endoscope is actuated so that it captures the surgical scene from the same perspective as the surgeon. The surgeon wears a head-mounted display to observe the endoscopic video. The frustum of the flexible endoscope is rendered as an augmented reality overlay to provide surgical guidance. We developed the prototype, FlexiVision, integrating a 6-DOF robotic flexible endoscope based on the da Vinci Research Kit and Microsoft HoloLens. We evaluated the proposed automatic control method via a lesion observation task, and evaluated the AR surgical guidance in a lesion targeting task. The multi-user study results demonstrated that, for both tasks, FlexiVision significantly reduced the completion time (by 59% and 58%), number of errors (by 75% and 95%) and subjective task load level. With FlexiVision, the flexible endoscope could act as the surgeon’s eyes teleported into the abdominal cavity of the patient.