Semantic Joint Monocular Remote Sensing Image Digital Surface Model Reconstruction Based on Feature Multiplexing and Inpainting

Digital surface model (DSM) presents height information of the Earth’s surface and plays an important role in many remote sensing (RS) applications. Since the conventional acquisition of DSM is laborious and expensive, DSM reconstruction from monocular RS images has attracted extensive research in recent years, which is an ill-posed problem and thus rather challenging. Related works have achieved great accomplishments in this regard; however, they still face some limitations in training robustness, accuracy, and efficiency. To address the issues, a semantic joint monocular RS image DSM regression framework is proposed in this article, whose salient points include that: 1) semantic segmentation is integrated into the DSM regression task so that a shared backbone can extract complementary features from each objective to improve the performance of the individual task. Meanwhile, based on the consistency of the two training objectives, a two-stage joint loss function is introduced to improve the convergence and robustness of model training; 2) an encoding–decoding backbone is designed based on feature multiplexing, which simultaneously achieves multiscale feature fusion and information decoupling, thereby greatly reducing model parameters and improving efficiency while ensuring feature extraction effect; and 3) an iterative upsampling approach is introduced to transform the full-scale spatial features into large receptive-field and locally discriminative dynamic kernels, which are used to inpaint coarse-grained features while decoding, thus enhancing regression accuracy. Finally, experiments demonstrate the effectiveness of the proposal. It is easy to train and achieves superior or comparable accuracy compared with state-of-the-art related works while improving the efficiency by a large margin.