Social image refinement and annotation via weakly-supervised variational auto-encoder

Abstract The ever-increasing size of social images and their corresponding imperfect labels have made social image refinement and annotation a crucial problem in supervised learning. However, previous models based on nearest neighbors or matrix completion are limited when the social image set is huge and labels are highly sparse. Deep generative models utilize inference and generative networks to infer latent variables by introducing an observed data variable; they can handle imperfect data, capture noisy data, and fill in missing data variables. In this paper, we propose a new social image refinement and annotation model based on the weakly-supervised variational auto-encoder generative model. First, we formulate the social image refinement and annotation problem as a joint distribution of social images and labels in a probabilistic generative model. Secondly, we derive a new evidence lower bound object to handle imperfect labels. Thirdly, we design a new multi-layer neural network including inference and generative networks to optimize the new evidence lower bound efficiently. Finally, we perform a comparison of our model with other representative models on several real-world social image datasets. Experimental results on social image refinement and annotation tasks show that the proposed model is competitive or even better than existing state-of-the-arts.