ISETAuto: Detecting vehicles with depth and radiance information

Autonomous driving applications use two types of sensor systems to detect vehicles - depth sensing LiDAR and radiance sensing cameras. We compare the performance (average precision) of a ResNet for vehicle detection in complex, daytime, driving scenes when the input is a depth map [D = d(x,y)], a radiance image [L = r(x,y)], or both [D,L]. (1) When the spatial sampling resolution of the depth map and radiance image are both equal to typical camera resolutions, a ResNet detects vehicles at higher average precision from depth than radiance. (2) When the spatial sampling of the depth map matches the range of current LiDAR devices, the average precision is higher for radiance than depth. (3) A hybrid system that combines depth and radiance has higher average precision than systems using depth or radiance alone. We confirm these observations in both simulation and real-world data. We explain the advantage of combining depth and radiance by noting that the two types of information have complementary weaknesses. The radiance data are limited by dynamic range, motion blur and illumination variation; the LiDAR data have low spatial resolution. The ResNet effectively combines the two data sources to improve vehicle detection.