Remote-sensing image retrieval with tree-triplet-classification networks

Abstract For the past few years, convolutional neural networks (CNNs) have played a dominant role in content-based remote sensing image retrieval (CBRSIR) because of their markedly superior performance. However, most of the CNN models used for CBRSIR were originally meant for image classification instead of for image retrieval. We argue that triplet networks designed in the context of metric learning are more natural and suitable for CBRSIR. However, they only use information about whether or not two input images belong to the same class, and fail to fully exploit class labels. Besides, all existing CNN-based CBRSIR methods ignore prior knowledge about interclass relationship, which, if used properly, can greatly improve retrieval performance. To address these issues, we introduce an easy way to organize semantic relationship among classes as a category tree, and propose a novel CNN model called tree-triplet-classification (T-T-C) network, the key characteristics of which can be summarized as follows: firstly, a T-T-C network integrates metric learning with classification prediction, simultaneously learning similarity measurement and categorizing images and, hence, taking advantage of the complementary capabilities of the existing CNN-based approaches; secondly, the loss functions coupled with a T-T-C network lay emphasis on the structure of a category tree, using prior semantic knowledge to adaptively adjust the “pull-push” mechanism during training; finally, a T-T-C network is lightweight, and its features are very compact. We carry out extensive experiments over publicly available datasets, and achieve a state-of-the-art retrieval performance.