Multi-sensor land cover classification with sparsely annotated data based on Convolutional Neural Networks and Self-Distillation

Extensive research studies have been conductedin recent years to exploit the complementarity among multi-sensor (or multi-modal) remote sensing data for prominentapplications such as land cover mapping. In order to make a step further with respect to previous studies which investigate multi-temporal SAR and optical data or multi-temporal/multi-scale optical combinations, here we propose a deep learning framework that simultaneously combine all these input sources, specifically multi-temporal SAR/optical data and fine scale optical information. Our proposal relies on a patch-based multi-branch convolutional neural network (CNN) that exploits different per source CNN encoders to deal with the specificity of the input signals. Additionally, our framework is equipped with a self-distillation strategy to boost the per source analyses. This new strategy leverages the final prediction of the multi-source framework to guide the learning of the per source CNN encoders supporting the network to learn from itself.Experiments are carried out on two real world benchmarks, namely the Reunion island (a french overseas department) and the Dordogne study site (a southwest department in France)where the annotated reference data was collected under operational constraints (sparsely annotated ground truth data). Obtained results, providing an overall classification accuracyof about 94% (resp. 88%) on theReunion island(resp. theDordogne) study site. These findings highlight the effectivenessof our framework based on Convolutional Neural Networks andself-distillation to combine heterogeneous multi-sensor remotesensing data and confirm the benefit of multi-modal analysisfor downstream tasks such as land cover mapping.