A Zero-Shot Learning Framework via Cluster-Prototype Matching

Abstract Given the descriptions of classes, Zero-Shot Learning (ZSL) aims to recognize unseen samples by learning a projection between the visual features of samples and the semantic descriptions (prototypes) of classes from seen data. However, due to the inherent distribution gap between seen and unseen domains, the learned projection is generally biased to seen classes and may produce misleading relationships between unseen samples and prototypes (sample-prototype relationship). To tackle this problem, we propose a Cluster-Prototype Matching (CPM) framework which exploits the distribution information of samples to explore the cluster structure of samples and then use the robust cluster-prototype relationship to correct the biased sample-prototype relationship. Specifically, we first use an iterative cluster generation module to identify the underlying cluster structure of samples based on their embedding features, which are acquired via a basic ZSL model. Then each identified cluster will be matched with a specific class prototype through the Kuhn-Munkres algorithm, based on which we can export a sharp cluster-prototype similarity. Finally, the cluster-prototype similarity is combined with the sample-prototype similarity to determine the class labels of test samples. We apply CPM to five well-established ZSL methods and the experimental results show that CPM can significantly improve the performance of basic models and enable them achieve or beyond the state-of-the-art.