Cross Modal Few-Shot Contextual Transfer for Heterogenous Image Classification

The deep transfer learning aims at dealing with challenges in new tasks with insufficient samples. However, when it comes to few-shot learning scenario, due to the low diversity of several known training samples, they are prone to dominated by specificity, thus misleading to one-sidedness local features instead of the reliable global feature of actual categories they belonging to. To this end, contextual information is able to be chosen as supplement. In this paper, a cross-modal method is proposed to realize context awareness transfer in few-shot image classification scene, which fully utilizes the information in heterogeneous data. The similarity measure in image classification task is reformulated that the important semantic information, including inherent one in textual modal and extracted one from image, with visual feature information from object in background are taken into account together as supplement to inhibit the effect of sample specificity. On this basis, the deep transfer scheme is also used in reusing a powerful extractor from well pre-trained model for better recognizing local visual features and reorganizing the feature recognition pattern in the convolutional layers. Simulation experiments show that the introduction of cross-modal and intra-modal contextual information can effectively suppress the deviation of defining category features under few samples, and improve the accuracy of few-shot image classification task.