On the relationship between click rate and relevance for search engines

Evaluation of search engine result relevance has traditionally been an expensive process done by human judges. Researchers have sought cheap automated proxies for such judgments. This paper examines the relationship between relative click rates(oftwoengines)andrelativehumanjudgmentsofresultsets returnedbythose engines. Previous work has indicated that human judgments are more consistent if provided in a relative form. We additionally observe that clicks are a function not only of the clicked result, but also of its competing neighborhood. These observations force an experimental design where we collect relative judgments of sets of results, rather than judgments on individual results. We conduct a large empirical study using forty judges, thousands of live users and hundreds of queries. Our results comparing Yahoo with another search engine in October 2003 show that in aggregate, higher click rate is indicative of higher relevance but the strength of the association is only moderate 40%. Qualitative analysis suggests the association is not stronger because users click for reasons other than relevance such as curiosity and confusion. However, there are classes of queries (such as navigational queries) for which click rates are good indicators of relevance.