A Fast RetinaNet Fusion Framework for Multi-spectral Pedestrian Detection

Abstract At present, the mainstream visible pedestrian detector is easily affected by the ambient lighting, the complex background, and the pedestrians distance. And the infrared images can compensate for this defect of visible images because of its insensitivity to illumination conditions. Based on the Deep Convolutional Neural Network (DCNN), we propose a multispectral pedestrian detector that combines visual-optical (VIS) image and infrared (IR) image. It is found that the detector improves the performance of pedestrian detection under weak illumination conditions by combining the complementary feature information of the two images. We have carefully designed three DCNN fusion architectures to study the better fusion stages of the two-branch DCNN. In addition, we compared the three fusion strategies and found that the sum fusion strategy shown better performance to our multispectral detector. Our multispectral pedestrian detectors are more adaptable to the around-the-clock applications, eliminating the need for adequate illumination conditions for applications such as autonomous driving and unattended monitoring. By testing on the public multispectral benchmark dataset KAIST, our best fusion architectures achieved a log-average miss rate of 27.60%, which is comparable to the recent state-of-the-art detector, our detector achieves half the time overhead while guaranteeing detection performance.