Joint Visual and Semantic Optimization for zero-shot learning

Abstract Zero-shot learning (ZSL) aims to classify instances whose classes could be unseen during training. Most existing ZSL methods project visual or semantic features into the space of the other one, or into a common subspace. The main goal of projection is to find out the similar features in the latent subspace. However, existing methods barely consider common features that preserve knowledge, here we refer to these features as the shared concepts, which are essential to model the relationship between the visual and semantic spaces. In this paper, we exploit the underlying concepts shared by both visual and semantic features in a latent common subspace and propose to match their latent visual and semantic representations. To reduce domain shift and information loss, we introduce reconstruction losses for both visual and semantic features. As a result, the reconstruction regularizations are added to the similar features and thereby obtain knowledge preserving shared concepts via the proposed method. Mathematically, it is formulated as the minimization problem for mutual orthogonal projection to their latent common subspace. The problem involves two projection variables, thus we develop an algorithm based on the Gauss–Seidel iteration scheme and split the problem into two subproblems in the scheme. These two subproblems are further solved by searching algorithms based on the Barzilai–Borwein stepsize. Extensive experiments on six benchmark data sets are conducted to demonstrate that the accuracy of the proposed method is better than that of existing ZSL methods.