Cryptographic verification of test coverage claims

The market for software components is growing, driven on the demand side by the need for rapid deployment of highly functional products, and on the supply side by distributed object standards. As components and component vendors proliferate, there is naturally a growing concern about quality, and the effectiveness of testing processes. White box testing, particularly the use of coverage criteria, is a widely used method for measuring the thoroughness of testing efforts. High levels of test coverage are used as indicators of good quality control procedures. Software vendors who can demonstrate high levels of test coverage have a credible claim to high quality. However, verifying such claims involves knowledge of the source code, test cases, build procedures etc. In applications where reliability and quality are critical, it would be desirable to verify test coverage claims without forcing vendors to give up valuable technical secrets. In this paper, we explore cryptographic techniques that can be used to verify such claims. Our techniques have some limitations; however, if such methods can be perfected and popularized, they can have an important leveling effect on the software market place: small, relatively unknown software vendors with limited resources can provide credible evidence of high-quality processes, and thus compete with much larger corporations.