Benchmarking in digital circuit design automation

This paper focuses on benchmarking, which is the main experimental approach to the design method and EDA-tool analysis, characterization and evaluation. We discuss the importance and difficulties of benchmarking, as well as the recent research effort related to it. To resolve several serious problems related to quality of benchmarking and use of practical industrial benchmarks, we proposed an adequate benchmarking methodology based on the statistical experimental design approach, and developed corresponding digital circuit benchmark generators. These benchmark generators enable research, evaluation and fine-tuning of circuit synthesis methods and EDA-tools largely independent of the actual industrial benchmarks, and much better than having only some industrial benchmarks. Using the results of extensive experiments that involved large sets of diverse benchmarks generated with our FSM benchmark generator, we discuss several crucial problems of benchmarking and demonstrate how to resolve them.