Single-Stream CNN With Learnable Architecture for Multisource Remote Sensing Data

In this article, we propose an efficient and generalizable framework based on a deep convolutional neural network (CNN) for multisource remote sensing (RS) data joint classification. While recent methods are mostly based on multistream architectures, we use group convolution (GConv) to construct equivalent network architectures efficiently within a single-stream network. Based on a recent technique called dynamic grouping convolution (DGConv), we further propose a network module named separable DGConv (SepDGConv), to make GConv hyperparameters, and, thus, the overall network architecture, learnable during network training. In the experiments, the proposed method is applied to residual network (ResNet) and UNet, and the adjusted networks are verified on three very diverse benchmark datasets (i.e., Houston2018 data, Berlin data, and MUUFL Gulfport Hyperspectral and LiDAR Airborne Data Set (MUUFL) data). Experimental results demonstrate the effectiveness of the proposed single-stream CNNs, and in particular, SepG-ResNet18 improves the state-of-the-art classification overall accuracy (OA) on hyperspectral–synthetic aperture radar (HS–SAR) Berlin dataset from 62.23% to 68.21%. In the experiments we have two interesting findings. First, using DGConv generally reduces test OA variance. Second, multistream is harmful to model performance if imposed to the first few layers, but becomes beneficial if applied to deeper layers. Altogether, the findings imply that the multistream architecture, instead of being a strictly necessary component in deep learning models for multisource RS data, essentially plays the role of model regularizer. Our code is publicly available at https://github.com/yyyyangyi/CNNs-for-Multi-Source-Remote-Sensing-Data-Fusion . We hope our work can inspire novel research in the future.