Pyramid Fully Convolutional Network for Hyperspectral and Multispectral Image Fusion

Low spatial resolution hyperspectral (LRHS) and high spatial resolution multispectral (HRMS) image fusion has been recognized as an important technology for enhancing the spatial resolution of LRHS image. Recent advances in convolutional neural network have improved the performance of state-of-the-art fusion methods. However, it is still a challenging problem to effectively explore the spatial information of HRMS image. In this paper, we propose a pyramid fully convolutional network made up of an encoder sub-network and a pyramid fusion sub-network to address this issue. Specifically, the encoder sub-network aims to encode the LRHS image into a latent image. Then, this latent image, together with a HRMS image pyramid input, is used to progressively reconstruct the high spatial resolution hyperspectral image in a global-to-local manner. Furthermore, to sharpen the blurry predictions easily obtained by the standard $l_{2}$ loss, we introduce the gradient difference loss as a regularization term. We evaluate the proposed method on three datasets acquired by three different satellite sensors. Experimental results demonstrate that the proposed method achieves better performance than several state-of-the-art methods.