A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace.

Purpose Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales; the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. Method For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the EPAs across clerkships. The WBA scored students' performance using both the Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi coefficient was used to estimate reliability; a cut-off of at least 0.70 was used to conduct D-studies. Results Using the Ottawa scale, variability attributable to the student ranged from 0.8%-6.5%. For the Chen scale, student variability ranged from 1.8%-7.1%. This indicates the majority of the variation was due to the rater (42.8%-61.3%) and other unexplained factors. A range from 28 to 127 assessments were required to obtain a Phi coefficient of 0.70. For 2 EPAs, utilizing faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. Conclusions Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for EPAs.