Discriminatively-learned global image representation using CNN as a local feature extractor for image retrieval

This work introduces an image retrieval framework based on using deep convolutional neural networks (CNN) as a local feature extractor. Motivated by the great success of CNN in recognition tasks, one may be tempted to simply adopt the output of CNN as a global image representation for retrieval. This straightforward approach, however, has proved deficient, because it can be vulnerable to various image transformation attacks. To address this issue, we propose to treat CNN as a local feature extractor, and a local image patch selection mechanism is developed to extract discriminative patches by observing their objectness responses, aspect ratios, relative scales, and locations in the image. The criterion is given by a learned posterior probability indicating how likely the image patch in question will find a correspondence in another similar image. In addition, the CNN's weight parameters are specifically adapted by a contrastive loss function to suit retrieval tasks. Extensive experiments on typical retrieval datasets confirm the superiority of the proposed scheme over the state-of-the-art methods.