Mutating Code Annotations: An Empirical Evaluation on Java and C# Programs

Abstract Mutation testing injects code changes to check whether tests can detect them. Mutation testing tools use mutation operators that modify program elements such as operators, names, and entire statements. Most existing mutation operators focus on imperative and object-oriented language constructs. However, many current projects use meta-programming through code annotations. In a previous work, we have proposed nine mutation operators for code annotations focused on the Java programming language. In this article, we extend our previous work by mapping the operators to the C# language. Moreover, we enlarge the empirical evaluation. In particular, we mine Java and C# projects that make heavy use of annotations to identify annotation-related faults. We analyzed 200 faults and categorized them as “misuse,” when the developer did not appear to know how to use the code annotations properly, and “wrong annotation parsing” when the developer incorrectly parsed annotation code (by using reflection, for example). Our operators mimic 95% of the 200 mined faults. In particular, three operators can mimic 82% of the faults in Java projects and 84% of the faults in C# projects. In addition, we provide an extended and improved repository hosted on GitHub with the 200 code annotation faults we analyzed. We organize the repository according to the type of errors made by the programmers while dealing with code annotations, and to the mutation operator that can mimic the faults. Last but not least, we also provide a mutation engine, based on these operators, which is publicly available and can be incorporated into existing or new mutation tools. The engine works for Java and C#. As implications for practice, our operators can help developers to improve test suites and parsers of annotated code.