Visual relationship detection with region topology structure

Abstract Visual relationship detection is crucial for scene understanding of images, which aims to detect objects in the image and classify the visual relation for each pair of objects. In addition, it builds a bridge between computer vision and natural language. Current studies transform the visual relationship detection task into a classification task between detected objects, and propose various models which integrate vision feature, position feature and semantic features of certain image regions. Following this manner, our proposed method also extracts features from these three cues. Furthermore, It is often neglected that the topology between the regions plays an important role in capturing different visual relationships. In order to take advantage of topology, we treat the region visual features as vertexes and construct a visual region graph, and we model the dependency between ( subject , object ) pairs with the weighted edges. We further propose a visual relation detection framework based on the regional topology structure, which enables the model to incrementally aggregate topology structure information. We evaluate our method on VRD dataset and VG dataset, the results of the proposed method are close to even higher than that of the state-of-art methods on some evaluation metrics.