Fusion of stereoscopic video and laparoscopic ultrasound for minimally invasive partial nephrectomy

The development of an augmented reality environment that combines laparoscopic video and ultrasound imaging for image-guided minimally invasive abdominal surgical procedures, such as partial nephrectomy and radical prostatectomy, is an ongoing project in our laboratory. Our system overlays magnetically tracked ultrasound images onto endoscopic video to create a more intuitive visualization for mapping lesions intraoperatively and to give the ultrasound image context in 3D space. By presenting data in a common environment, this system will allow surgeons to visualize the multimodality information without having to switch between different images. A stereoscopic laparoscope from Visionsense Limited enhances our current system by providing surgeons with additional visual information through improved depth perception. In this paper, we develop and validate a calibration method that determines the transformation between the images from the stereoscopic laparoscope and the 3D locations of structures represented by a tracked laparoscopic ultrasound probe. We first calibrate the laparoscope with a checkerboard pattern and measure how accurate the transformation from image space to tracking space is. We then perform a target localization task using our fused environment. Our initial experience has demonstrated an RMS registration accuracy in 3D of 2.21mm for the laparoscope and 1.16mm for the ultrasound in a working volume of 0.125m 3 , indicating that magnetically tracked stereoscopic laparoscope and ultrasound images may be appropriately combined using magnetic tracking as long as steps are taken to ensure that the magnetic field generated by the system is not distorted by surrounding objects close to the working volume.