No evidence amalgamation without evidence measurement

In this paper we consider the problem of how to measure the strength of statistical evidence from the perspective of evidence amalgamation operations. We begin with a fundamental measurement amalgamation principle (MAP): for any measurement, the inputs and outputs of an amalgamation procedure must be on the same scale, and this scale must have a meaningful interpretation vis a vis the object of measurement. Using the p value as a candidate evidence measure, we examine various commonly used approaches to amalgamation of evidence across similar studies, including standard forms of meta-analysis. We show that none of these methods satisfies MAP. Thus an underlying measurement problem remains. We argue that a successful approach to evidence amalgamation necessitates a solution to the problem of evidence measurement, and we suggest some lines of reasoning that might guide further work towards this end.