3D-Reconstruction and Semantic Segmentation of Cystoscopic Images

Bladder cancer (BCa) is the fourth most common cancer and the eighth most common cause of cancer-related mortality in men. Although roughly 75% of patients are diagnosed with non-muscle invasive bladder cancer (NMIBC), recurrence rates are high even at a low stage and grade. Transurethral resection (TURB) is required for establishing the pathological diagnosis and clinical staging of patients. In daily clinical practice, however, conventional tumor documentation after TURB lacks accuracy, posing a major limitation for patient follow-up and surveillance. Novel technologies to facilitate data documentation and interpretation processes are imperative to maximize patients’ outcomes. As part of the RaVeNNA-4pi initiative, our contribution is twofold: first, we propose a bladder 3D-reconstruction method using Structure-from-Motion (SfM). Second, we propose deep convolutional neural networks (DCNN) for cystoscopic image segmentation to improve the interpretation of cystoscopic findings and localization of tumors. 3D reconstruction of endoscopic images assists physicians in navigating the bladder and monitoring successive resections. Nevertheless, this process is challenging due to an endoscope’s narrow field of view (FoV), illumination conditions and the bladder’s highly dynamic structure. So far in our project, the SfM approach has been tested on a bladder phantom, demonstrating that the processing sequence permits a 3D reconstruction. Subsequently, we will test our approach on bladder images from patients generated in real-time with a rigid cystoscope. In recent years, deep learning (DL) has enabled significant progress in medical image analysis. Accurate localization of structures such as tumors is of particular interest in processing medical images. In this work, we apply a DCNN for multi-class semantic segmentation of cystoscopic images. Moreover, we introduce a new training dataset for evaluating state-of-the-art DL models on cystoscopic images. Our results show that on average a 0.67 Dice score coefficient (DSC) can be achieved.