An Industrial Evaluation of Test Prioritisation Criteria and Metrics

Automated production systems become more and more complex, which makes it increasingly difficult for testers to keep track of performed changes and executed test cases. The risk of missing important test cases while planning the test execution is high, especially for testers with little experience. An automatic test prioritisation based on metrics supports the tester in selecting the right test cases for the test execution. In industry, many different test prioritisation criteria and strategies are used for this purpose. In an industrial interview, experts discussed and ranked prioritisation criteria that are currently used within the respective companies. As a result, this paper presents the cactus prioritisation model, which graphically resembles the industrial ranking and weighting of the criteria. Based on the prioritisation cactus and its criteria, a simple prioritisation metric is introduced to determine the utility of each test case regarding the tested system. The test cases are prioritised according to their descending utility. Furthermore, approaches and metrics to realise the different individual prioritisation criteria are proposed.