An FPGA Implementation of Real-Time Object Detection with a Thermal Camera

We demonstrate a sparse YOLOv2-based object detector with a thermal camera. A thermal camera outputs pixel values which represent heat (temperature), and the output is gray-scale images. Since the thermal cameras do not depend on whether there is the light or not unlike other visible range cameras, object detection using the thermal camera is reliable without ambient surrounding. This topic is of a broad interest in object surveillance and action recognition. However, since it is challenging to extract informative features from the thermal images, the implementation challenges of the object detector with high accuracy remain. In recent works, convolutional neural networks (CNNs) outperform conventional techniques, and a variety of object detectors based on the CNNs have been proposed. The representative networks are single-shot detectors that consist of a CNN and infer locations and classes simultaneously (e.g., SSD and YOLOv2). Although the primary advantage of the type is that it enables to train detection and classification simultaneously, the resulting increased computation time and area requirements can cause problems of implementation on an FPGA. Also, as for the proposed networks on RGB three channel images, one of the problems is false positive; the realization of more reliable object detector is required. To realize a real-time reliable object detector, we investigate an FPGA implementation of a sparse YOLOv2-based one whose inputs are four-channel images that consist of both the RGB and the thermal ones. In this demonstration, we show a performance comparison between an RGB-based detector and our proposed one on FPGAs.