3D Volume Reconstruction from Single Lateral X-Ray Image via Cross-Modal Discrete Embedding Transition

In this paper, we propose a deep neural network (DNN)-based volumetric image reconstruction framework, enabling an end-to-end volume inference from a 2D X-ray image. The proposed framework is built upon the vector quantised-variational autoencoders of the 2D X-ray images and the 3D cone-beam CT (CBCT) images. We present a bi-directional cross-modal transition module between the discrete latent variables of the 2D and 3D images. In the X-ray-to-CBCT pass, the transition module takes the feature maps of the X-ray images as an input and outputs the discrete volumetric embedding, which is combined with the codebook in the embedding space for the reconstruction of CBCT images. On the other hand, the CBCT-to-X-ray pass realizes the DNN-based volume rendering. Given real clinically-obtained X-ray images without paired volumetric images, we devise an unsupervised learning scheme to optimize the discrete embedding transition module. Our approach utilizes the quantised latent representation and the cross-modality transition module to infer volumetric images. The proposed method has been applied to the lateral X-ray-based CBCT image reconstruction in clinical orthodontics, achieving performance improvements over compared methods.