Evaluating second-order probability judgments with strictly proper scoring rules

Empirical studies have demonstrated that uncertainty about event probabilities, also known as ambiguity or second-order uncertainty, can affect decision makers’ choice preferences. Despite the importance of second-order uncertainty in decision making, almost no effort has been directed towards the development of methods that evaluate the accuracy of second-order probabilities. In this paper, we describe conditions under which strictly proper scoring rules can be used to assess the accuracy of second-order probability judgments. We investigate the effectiveness of using a particular strictly proper scoring rule the ranked probability score - to discourage biased assessments of second-order uncertainty.