Measuring Unit Test Accuracy

Unit testing is widely adopted in software development to automate verification of individual functions. Likewise, computer science curricula are increasing their emphasis on teaching unit testing skills. However, coverage and other measurements of test quality have limitations and may even encourage poor practices when used as an assessment and/or feedback mechanism for students. In this paper, we introduce test accuracy as a measurement that compares how well unit tests perform at distinguishing acceptable from unacceptable function implementations. To study measurements of test quality, we evaluated an assignment where students (n=103) developed a class along with accompanying tests for each of its functions. Our quantitative analysis compares test accuracy with coverage and bug identification measurements by examining each of their relationships with a lack of bugs in the students' implementations. Of the three measurements, accuracy had the strongest positive correlation (p=.648, p