A Case for Software Specific Natural Language Techniques

For over two decades, software engineering (SE) researchers have been importing tools and techniques from information retrieval (IR). Initial results have been quite positive. For example, when applied to problems such as feature location or re-establishing traceability links, IR techniques work well on their own, and often even better in combination with more traditional source code analysis techniques such as static and dynamic analysis. However, recently there has been growing awareness among SE researchers that IR tools and techniques are designed to work under a different set of assumptions than those that hold for a software system. Thus it may be beneficial to consider IR inspired tools and techniques that are specifically designed to work with software. One aim of this work is to provide quantitative empirical evidence in support of this observation. To do so a new technique is introduced that captures the level of difficulty found in an information need, the true, often latent, information that a searcher desires to know. The new technique is used to compare two test collections: the natural language TREC 8 collection and the software engineering JabRef collection. Analysis of the data leads to three significant findings. First, the variation in difficulty of the SE information needs is much larger than that of the natural language information needs, second, the most challenging of the SE information needs is far easier than the least challenging of the natural language information needs, and finally, variations of the queries used to uncover a latent information need have far less impact in the natural language collection than in the software engineering collection.