Towards a Benchmark for Evaluating Design Pattern Miner Tools

Recovering design pattern usage in source code is a very difficult task. Several tools are described in the literature for this purpose, but there is little work invested in evaluating them. The main reason for this is the lack of an approved benchmark for these tools. In this paper we present work in progress towards creating a benchmark, called DEEBEE (design pattern evaluation benchmark environment), for evaluating and comparing design pattern miner tools. It is programming language, tool, pattern and software independent, and it is open to the community and freely available. Currently, the benchmark database contains the results of three tools: Columbus (C++), Maisa (C++), and design pattern detection tool (Java). The tools were evaluated on reference implementations of patterns and on open source software (Mozilla, NotePad++, JHotDraw, JRefactory and JUnit). Additionally, instances recovered by researchers are added from NotePad++ as well. Some recovered patterns are already verified by experienced developers. This work is the first step in building a large reference database of design pattern usage in open source software and we expect that researchers will join us in this effort.