An empirical assessment of baseline feature location techniques

Feature Location (FL) aims to locate observable functionalities in source code. Considering its key role in software maintenance, a vast array of automated and semi-automated Feature Location Techniques (FLTs) have been proposed. To compare FLTs, an open, standard set of non-subjective, reproducible “compare-to” FLT techniques (baseline techniques) should be used for evaluation. In order to relate the performance of FLTs compared against different baseline techniques, these compare-to techniques should be evaluated against each other. But evaluation across FLTs is confounded by empirical designs that incorporate different FL goals and evaluation criteria. This paper moves towards standardizing FLT comparability by assessing eight baseline techniques in an empirical design that addresses these confounding factors. These baseline techniques are assessed in twelve case studies to rank their performance. Results of the case studies suggest that different baseline techniques perform differently and that VSM-Lucene and LSI-Matlab performed better than other implementations. By presenting the relative performances of baseline techniques this paper facilitates empirical cross-comparison of existing and future FLTs. Finally, the results suggest that the performance of FLTs partially depends on system/benchmark characteristics, in addition to the FLTs themselves.