Pan-Sharpening via Deep Locally Linear Embedding Residual Network

The goal of pan-sharpening tasks is to fuse panchromatic (PAN) images and low-spatial-resolution (LR) multispectral (MS) images for the purpose of aggregating texture and spectral information. Although traditional embedding-based pan-sharpening methods achieve competitive results, they are limited by the shallow network and not suitable for large-scale datasets. In this study, we design a novel multiscale locally linear embedding residual network (LLERN) that consists of two phases: the spectral preservation phase and the structural preservation phase. As the pretreatment of the structural preservation network, the spectral preservation network aims to upscale the LR MS image while retaining spectral information. The proposed locally linear embedding residual block (LLERB) in the structural preservation phase can search for similar sparse patches from the PAN image space and embed the corresponding local geometric relationship into the residual space to enhance the MS image. Extensive experiments suggest that the proposed LLERN outperforms state-of-the-art methods from visual and quantitative perspectives, and confirm the assumption that LR image patches and residual image patches in a local region share a similar manifold structure, which can be used to guide deep-learning modeling with improved interpretability. The source code is available at https://github.com/jiaming-wang/LLERN .