Development of an Implementation-Agnostic Quality Metric for Evaluating the Accuracy of Position Estimations from Vision-Based Navigation Algorithms

As the focus in the field of navigation increasingly shifts toward alternatives to Global Navigation Satellite System (GNSS) aiding, vision-based navigation (VBN) techniques have proven to be especially promising. The error characteristics of VBN systems are not currently well understood and can vary significantly from system to system. A quality metric is needed in order for a navigation system to evaluate the accuracy of VBN position estimates for inclusion in the overall navigation solution and to aid in navigation algorithm design and sensor fusion. An implementation-agnostic metric allows for direct comparisons between measurement sources. In this paper, a feature tracking algorithm-agnostic vision-based navigation quality metric is devised that uses a common set of variables to evaluate the expected accuracy of a VBN solution without any knowledge of the VBN algorithm being assessed. The quality metric algorithm can therefore be rapidly integrated with any feature tracking VBN sensor or system regardless of its underlying mechanization, reducing the effort required to implement the metric algorithm on fielded systems and allowing for the comparison and fusion of measurements from two or more unique VBN sensors. To aid in the development of the VBN metric algorithm, a baseline georeferenced VBN was designed and implemented in a Monte Carlo simulation using satellite imagery from the National Resource Conservation Service (NRCS) database. The data set contained images that varied in terrain, vehicle height, camera resolution, and camera pose uncertainty. For each Monte Carlo set, the median VBN position error was collected and categorized as either above or below an arbitrary accuracy in meters. This data set was then used to train multiple machine learning models ranging in complexity from linear ordinary least squares to various forms of classification trees, with the goal being to correctly categorize the expected error of the VBN measurements. Multiple combinations of VBN input variables were compared in the models to determine which variables most influenced the accuracy of a given VBN position estimate, with the goal being to train a machine learning algorithm to accurately predict VBN position error with the minimum number of inputs and without over-fitting any single data set. While the least-squares method performed reasonably well, the more sophisticated classification tree topologies proved best able to predict VBN position estimate accuracy using a combination of four variables: pitch/roll uncertainty, yaw uncertainty, vehicle height, and the pixel distance between identified features in the image. The performance of the quality metric was verified using an additional data set created from the NRCS database, as well as an independent flight test data set using a different VBN system. The quality metric algorithm was able to accurately categorize the expected VBN position estimate accuracy for approximately 90% of the VBN estimates generated from the simulated and flight test data sets; comparable performance was seen across the different types of classification tree topologies.