Off-vehicle evaluation of camera-based pedestrian detection

Performance evaluation and comparison of vision-based automotive modules is a growing need in automotive industry. Off-vehicle evaluation, using a database of video streams offers many advantages over on-vehicle evaluation in terms of reduced costs, repeatability and the ability to compare different modules under the same conditions. An off-vehicle evaluation platform for camera based pedestrian detection is presented, enabling evaluation of industrial modules and internally developed algorithms. In order to maintain a single video database despite variability in camera location and internal parameters, experiments were done with video warping techniques, in which a video is warped to look as if taken from a target camera. To obtain ground truth annotation, both manual and Lidar-based methods were tested. Lidar-based annotation was shown to achieve detection rate > 80% without human intervention, which can go up to 97.5% using a semi-supervised methodology with moderate human effort. Finally, we examined several performance metrics, and found that the image-based detection criteria used in most of the literature does not fit certain automotive application well. A modified criterion based on real world coordinates is suggested.