Triangulated Rank-ordering of Web domains

The relative popularity of web sites, as expressed in published rankings, is of fundamental value in many contexts including search, advertising and research. In this paper, we consider the surprisingly challenging problem of generating consistent and reliable web site rankings based on unique visitors per day. We illustrate the challenge this represents using data from three large and independently-sourced Internet user panels. We begin by showing that generating a website ranking based simply on the observed unique daily visitors produces highly inconsistent rankings–even among the most popular sites. To mitigate the problems of bias and measurement error, we introduce a general methodology that identifies “canonical panelists”: an abstract class of user that exhibits consistent behavior across panels. Our definition is based on the epistemological technique of triangulation, which refers to observing the same object from multiple perspectives at the same moment in time. We show that panelists in the canonical class exhibit desirable characteristics including improved persistence. Most significantly, we show that defining a domain’s rank as a function of the aggregate behavior of canonical panelists improves overall alignment of rankings across all three of our panels.