Personalized Annotation for Photos with Visual Instance Search

Emotional and memorable moments are usually kept and shared on different online services such as Facebook, Flickr, Instagram, and Google Photos. As a result, one of users’ practical needs is to have their photos annotated automatically, especially with personalized tags. This motivates the authors to propose a system that can suggest personalized annotations for a photo uploaded to online services. Our system provides 2 major features. First, the system automatically recommends personalized annotations for newly uploaded photos based on visually similar photos uploaded in the past. Second, our system propagates manual annotations of users to other similar photos existed in their albums. To evaluate the performance of our system, we use the Oxford 5K Building Dataset and our own dataset consisting of personal photos collected from Facebook. Our systems achieves the mean Average Precision of 0.844 and 0.749 respectively on these two datasets. This demonstrates that our proposed solution can be potentially integrated as a useful utility or extension for online photo sharing services.