Applying Consensus-Based Measurement to the Assessment of Emerging Domains

Abstract : Situational judgment tests have been developed in the fields of Industrial/Organizational and Cognitive Psychology to predict performance and to evaluate theories of cognition. Production of these scales has usually required the opinions of subject matter experts to produce scoring keys or criterion data to compute empirically based standards. A simpler, more elegant procedure is considered that allows examinee responses to be scored as deviations from the consensus defined by the response distributions of the examinee sample. This approach is termed "Consensus Based Measurement" and has been applied to validate scales in "soft" knowledge domains, such as Emotional Intelligence, that lack certified experts and well-specified, objective knowledge. Data are summarized demonstrating substantial convergence between situational judgment test scores computed using expert and examinee-based scoring standards for which substantial expert and examinee data are available. The convergence indicates that examinee response distributions may be used to score situational judgment tests when expert responses are not available. Validity data for situational judgment scales that are scored with this approach are summarized.