Point Clouds-Image Fusion by Convolutional Method for Data Integrity using Neural Network

Recently, image-based deep learning models[1,2,3,4] have achieved state-of-art performance in classification and segmentation tasks. End-to-end deep learning models based on point clouds data led by PointNet[5] also achieve good object detection effect. At present, cameras, LIDARs and other sensors have become the main sources of information extraction from the real world. However, image data do not have accurate depth information, point clouds data is sparse without color information? using single sensor data can no longer meet the increasingly complex and unknown challenges. In this paper, a point clouds-image fusion method based on convolutional method is proposed. Through convolution and deconvolution operation, two kinds of data can better compensate, and the integrity of data can be improved, so that it can be better applied in complex and unknown environment. To further highlight the benefits of using a multi-sensor fusion method for object detection, a data set scenes was extracted from driving sequences of the KITTI[6] raw data set. As excepted, Fusioned data achieved expected performance in object detection task.