Automation and consistency analysis of test cases written in natural language: An industrial context

Abstract We present here a novel test automation strategy that receives as input a freestyle natural language (NL) test case (consisting of a sequence of test steps) and produces executable test scripts. This strategy relies on a database of previously automated seed test steps, available for reuse. New steps are automated via a capturing process by a tester, without requiring any programming knowledge. Automated tests can be executed by a replay facility. We discuss the reuse improvement, implementation effort, and user feedback regarding the industrial applicability and usability of our capture & replay tool. We then show that restricting the input textual description to obey a proposed Controlled NL (CNL) brings significant advantages: (1) reuse improvement; (2) the possibility of integration with a test generation framework; and (3) definition of consistency notions for test actions and test action sequences, that ensure, respectively, well-formedness of each action and a proper configuration to safely execute a sequence of actions. We formalize these consistency notions in Alloy and use the Alloy Analyzer to carry out the consistency check; the scalability of the analysis is assessed via an evaluation considering a repository with real test cases; the practical context of our work is mobile device testing, involving a partnership with Motorola Mobility, a Lenovo company.