Combining Visual Cues with Interactions for 3D–2D Registration in Liver Laparoscopy

Augmented Reality (AR) in monocular liver laparoscopy requires one to register a preoperative 3D liver model to a laparoscopy image. This is a difficult problem because the preoperative shape may significantly differ from the unknown intraoperative shape and the liver is only partially visible in the laparoscopy image. Previous approaches are either manual, using a rigid model, or automatic, using visual cues and a biomechanical model. We propose a new approach called the hybrid approach combining the best of both worlds. The visual cues allow us to capture the machine perception while user interaction allows us to take advantage of the surgeon’s prior knowledge and spatial understanding of the patient anatomy. The registration accuracy and repeatability were evaluated on phantom, animal ex vivo and patient data respectively. The proposed registration outperforms the state of the art methods both in terms of accuracy and repeatability. An average registration error below the 1 cm oncologic margin advised in the literature for tumour resection in laparoscopy hepatectomy was obtained.