Consistency of choice in nonparametric multiple comparisons

In this paper, we are interested in the inconsistencies that can arise in the context of rank-based multiple comparisons. It is well known that these inconsistencies exist, but we prove that every possible distribution-free rank-based multiple comparison procedure with certain reasonable properties is susceptible to these phenomena. The proof is based on a generalisation of Arrow's theorem, a fundamental result in social choice theory which states that when faced with three or more alternatives, it is impossible to rationally aggregate preference rankings subject to certain desirable properties. Applying this theorem to treatment rankings, we generalise a number of existing results in the literature and demonstrate that procedures that use rank sums cannot be improved. Finally, we show that the best possible procedures are based on the Friedman rank statistic and the k-sample sign statistic, in that these statistics minimise the potential for paradoxical results.