Specification coverage as a measure of test suite quality

This paper introduces specification coverage, a measure of test suite quality that indicates how much of a program’s behavior is exercised by a set of executions. Specification coverage complements code-based coverage metrics and may be better for certain tasks. Specification coverage is also complementary to specification-based test suite generation: it can be computed for any test suite and can be used without a priori knowledge of the program’s specification. This paper presents results from experiments that derive relationships between test suite size, specification coverage, code coverage (of various sorts), and bug detection. Relatively small test suites achieve high levels of specification coverage. Specification coverage is correlated with bug detection (when controlling for test suite size and code coverage). Specification coverage is correlated with code coverage (when controlling for test suite size). Finally, the paper discusses techniques for automatically producing specificationcoverage-complete test suites.