Thermal infrared and visible sequences fusion tracking based on a hybrid tracking framework with adaptive weighting scheme

Abstract Object tracking based on single sensor image sequences is now proved to be insufficient when facing complex challenging factors such as occlusions, background clutter, illumination variations, deformation and scale change. Complementary information between thermal infrared and visible image sequences is highly valuable and plays a critical role in tracking under complex scenarios. Previous fusion-before-tracking algorithms are not efficient and accurate enough due to the inevitable introduction of redundant information and considerable computational consumption. In this paper, we propose a robust fusion tracking method that exploits the abovementioned complementary information under a hybrid “tracking-by-detection” framework which consists of two tracking modules—the correlation filter based tracking (CFT) module and histogram based tracking (HIST) module. In CFT module, features extracted from both thermal infrared and visible images such as histogram of oriented gradient (HOG), image intensity and color names, are utilized to generate response maps and then adaptively fused through a denoising fusion scheme. In HIST module, a response map is obtained by adopting RGB color histogram in a statistical tracking model. Then, the response maps of two modules are fused via a new adaptive weighting scheme we proposed. Extensive experimental results on challenging thermal infrared and visible image sequences demonstrate the accuracy and robustness of the proposed method in comparison with several state-of-the-art methods.