Patch Relational Covariance Distance Similarity Approach for Image Ranking in Content-Based Image Retrieval

Content-Based Image Retrieval (CBIR) is an information retrieval framework for retrieving similar images based on objects in the images. Machine learning based CBIR consists of object detection, the majority of which rely on Convolutional Neural Network (CNN) as object detector, and image similarity ranking. However, object detection with CNN requires expensive retraining when new set of the images is added to the database, while current ranking techniques focus on visual characteristics without considering object's spatial information. In this work, we propose a new CBIR framework to alleviate the aforementioned problems. We employ the Hierarchical Deep Convolutional Neural Network (HD-CNN) for single object detection. HD-CNN has been shown to be more efficient in model retraining on partitions of large dataset. In addition, a new similarity measure based on the covariance descriptor called Patch Relational Covariance Distance Similarity (PRCDS) is proposed. PRCDS summarizes the low-level visual features as well as object's spatial information (patch arrangement descriptor) to rank the candidate images from the HD-CNN. Finally, the proposed framework was validated on a subset of ImageNet dataset, and the experimental results showed that the ranking based on the newly proposed similarity measure is consistent with human perception.