Semantic and edge-based visual odometry by joint minimizing semantic and edge distance error

Abstract In recent years, the progress made in deep learning for semantic segmentation has advanced development of semantic visual odometry (VO). Along with point-based and direct methods, VO has recently used edge features. However, mismatches are common in scenes in which the distribution of edges is complex owing to the lack of appropriate descriptors for edges at the present. In this paper, we propose a semantic-segmentation-aided edge-based VO (DSEVO). It is intended to improve the localization accuracy by decreasing mismatches in the edge alignment. In the reprojection process, the semantic and edge distance residual are considered to reduce the mismatches of edges between different frames. Then, camera motion estimation is accomplished by jointly minimizing the semantic and edge cost function. Our proposed method was evaluated on the public VKITTI and TUM RGB-D datasets. It was compared with state-of-the-art methods, including the respective feature-point-based, direct, and edge-based methods. We implemented a semantic-edge-based VO system. The experimental results showed that our method achieved the highest accuracy on most of the testing sequences.