CF3: Test Suite for Arithmetic Optimization of C Compilers

This article presents a compiler test suite “CF3,” which targets arithmetic optimization, especially constant folding, of C compilers. While Orange3 is a random test system which has detected bugs in the latest versions of GCC and LLVM/Clang, it may generate, for some compilers, many long error programs which detect the same bugs repeatedly. A new test suite CF3 is based on an obervation that most of the error programs which Orange3 found on GCCs have been reduced to small programs, each consisting of a single expression with 3 operations. CF3 consists of 10,985 files, which cover all the possible expression patterns build up with 3 operations, and each file contains 100 cases to test optimization for the expression pattern with different types and initial values of 4 variables. Experiments on GCC and LLVM/Clang show that CF3 detects more error program patterns than Orange3 within the same computation time.