If You Ask Nicely, I will Answer: Semantic Search and Today's Search Engines

Today's search engines are still very sensitive to the way queries are constructed. In some occasions, equivalent but slightly different forms of a query lead to completely different results. However, popular queries with only one right answer seem to be generally well served by search engines, which generally return the correct answer among their top 10 search results. Internet's redundancy of information and the recent proliferation of user generated content helps search engines to remain almost entirely keyword oriented and still robustly handle equivalent versions of queries. In this paper we propose a family of metrics to evaluate the semantical invariance of a given search engine, and we report experimental results for well-known engines such as Google, Yahoo!, Live and Ask.com, as well as for new semantic search engines like Hakia and Cuil.