Validation and verification of the remote agent for spacecraft autonomy

The six-day Remote Agent Experiment (RAX) on the Deep Space 1 mission will be the first time that an artificially intelligent agent will control a NASA spacecraft. Successful completion of this experiment will open the way for AI-based autonomy technology on future missions. An important validation objective for RAX is implementation of a credible validation and verification strategy for RAX that also "scales up" to missions that make full use of spacecraft autonomy. Autonomous flight software presents novel and difficult testing challenges that traditional flight software (FSW) does not face. Since autonomous software must respond robustly in an immense number of situations, the all-paths testing approaches used for traditional FSW is not feasible. Instead, we advocate a combination of scenario-based testing and model-based validation. This paper describes the testing challenges faced by autonomous spacecraft commanding software, discusses the testing strategies and model-validation methods that we found effective for RAX, and argues that these methods will "scale up" to missions that make full use of spacecraft autonomy. Among the key challenges for validating autonomous systems such as the RAX are ensuring adequate coverage for scenario-based tests, developing methods for specifying the expected behavior, and developing automated tools for verifying the observed behavior against those specifications. Another challenge, also faced by traditional FSW, is the scarcity of high-fidelity test-beds. The test plan must be designed to take advantage of lower-fidelity test-beds without compromising test effectiveness.