Robust fusion for RGB-D tracking using CNN features

Abstract Recently, RGB-D sensors have become popular. Many computer vision problems can be better dealt with depth data. It is a challenging problem to integrate depth data into a visual object tracker to address the problems such as scale change and occlusion. In this paper, we propose a robust fusion based RGB-D tracking method. Specifically, hierarchical convolutional neural network (CNN) features are first adopted to encode RGB and depth images separately. Next, target is tracked based on correlation filter tracking framework. Then the results of each CNN feature are localized according to the tracking results in a short period of time. Finally, the target is localized by jointly fusing the results of RGB and depth images. Model updating is finally carried out according to the differences between RGB and depth images. Experiments on the University of Birmingham RGB-D Tracking Benchmark (BTB) and the Princeton RGB-D Tracking Benchmark (PTB) achieve comparable results to state-of-the-art methods.