Robustness of spectrum-based fault localisation in environments with labelling perturbations

Abstract Most fault localisation techniques take as inputs a faulty program and a test suite, and produce as output a ranked list of suspicious code locations at which the program may be defective. If only a small portion of the executions are labelled erroneously, we expect a fault localisation technique to be robust to these errors. However, it is not known which fault localisation techniques with high accuracy are robust and which techniques are best at finding faults under the trade-off between accuracy and robustness. In this paper, a theoretical analysis of the impacts of labelling perturbations on spectrum-based fault localisation techniques (SBFL) is presented from different aspects first. We theoretically analyse the influence of labelling perturbations on three relations among risk evaluation formulas and the effect of mislabelling cases on the ranking of faulty statements. Then, we conduct controlled experiments on 18 programs with 3079 faulty versions from different domains to compare the robustness of 23 classes of risk evaluation formulas. Besides, experiments are conducted for evaluating the robustness of two neural network-based techniques. The impacts of perturbation degrees, number of faults and types of labelling perturbation on the robustness of formulas are empirically studied, and several interesting findings are obtained.