Manifold Matching with Application to Instance Search Based on Video Queries

In this paper we address the problem of matching video clips, each of which contains an instance from the same entity but undergoing transformation. To this end we formulate the problem as manifold matching by measuring the similarity between multiple manifolds, each represents a video clip. This work is novel in that it does not require a template or training. Instead it analyses the video by characterising the spatio-temporal information embedded in a frame sequence. Firstly the spatial Isomap is extended to spatio-temporal graph-based manifold embedding in order to discover the underlying structure of a video stream. Secondly linear models are extracted from each manifold through a hierarchical clustering method. The problem is then formulated as finding the distances between a pair of subspaces, each from one of the manifold. Experiment on Flicker dataset proved that the scheme was able to improve the search and retrieval performance over conventional approaches.