P-S Instance Retrieval via Early Elimination and Late Expansion

In daily life, it is common that viewers want to quickly browse scenes with their idols in TV series. In 2016, the TRECVID INS (Instance Search) task started to focus on identifying a specific target person in a target location. In this paper, we name this kind of task as P-S (Person-Scene) Instance Retrieval. As we know, most approaches handle this task by separately obtaining the person instance and the scene instance retrieval results, and directly combining them together. However, we find that the person and scene instance retrieval modules are not always effective at the same time, which will decrease the accuracy if the results are aggregated directly. To solve this problem, we attempt to achieve the results in two steps. (1) Early Elimination. There are many noisy data making person/scene instance retrieval score solely high, such as the occluded person or scene shots. Corresponding scores of these shots should be eliminated rather than calculated with noise. (2) Late Expansion. Considering the video»s continuity, person or scene in adjacent shots is likely to be the same one, hence we try to expand the results of those eliminated shots. On this basis, we propose an early elimination and late expansion method to improve the accuracy of P-S Instance Retrieval. Experimental results on the large-scale TRECVID INS dataset demonstrate the effectiveness of the proposed method.