Deep learning integral imaging for three-dimensional visualization, object detection, and segmentation

Abstract A depth slice image that is computationally reconstructed from an integral imaging system consists of focused and out of focus areas. The unfocused areas affect three-dimensional (3D) image analyses and visualization including 3D object detection, extraction, and tracking. In this work, we present a deep learning integral imaging system that can reconstruct a 3D image without the out of focus areas and can accomplish target detection and segmentation at the same time. A Mask-Regional Convolutional Neural Network (Mask-RCNN) deep learning algorithm was trained using a public dataset and applied to detect and segment multiple targets in two-dimensional (2D) elemental images in the integral imaging system. The 3D images were then reconstructed using segmented elemental images with the target detected. The proposed method works well in the presence of partial occlusions. Experimental results show the performance of the proposed scheme.