The wisdom of collective grading and the effects of epistemic and semantic diversity

A computer simulation is used to study collective judgements that an expert panel reaches on the basis of qualitative probability judgements contributed by individual members. The simulated panel displays a strong and robust crowd wisdom effect. The panel’s performance is better when members contribute precise probability estimates instead of qualitative judgements, but not by much. Surprisingly, it does not always hurt for panel members to interpret the probability expressions differently. Indeed, coordinating their understandings can be much worse.