Validation of the Omni: A Novel, Multimodality, and Longitudinal Surgical Skills Assessment

Objective The breadth of technical skills included in general surgery training continues to expand. The current competency-based training model requires assessment tools to measure acquisition, learning, and mastery of technical skill longitudinally in a reliable and valid manner. This study describes a novel skills assessment tool, the Omni , which evaluates performance in a broad range of skills over time. Design The 5 Omni tasks, consisting of open bowel anastomosis, knot tying, laparoscopic clover pattern cut, robotic needle drive, and endoscopic bubble pop, were developed by general surgery faculty. Component performance metrics assessed speed, accuracy, and quality, which were scaled into an overall score ranging from 0 to 10 for each task. For each task, ANOVAs with Scheffe's post hoc comparisons and Pearson's chi-squared tests compared performance between 6 resident cohorts (clinical years (CY1-5) and research fellows (RF)). Paired samples t-tests evaluated changes in performance across academic years. Cronbach's alpha coefficient determined the internal consistency of the Omni as an overall assessment. Setting The Omni was developed by the Department of Surgery at Duke University. Annual assessment and this research study took place in the Surgical Education and Activities Lab. Participants All active general surgery residents in 2 consecutive academic years spanning 2015 to 2017. Results A total of 62 general surgery residents completed the Omni and 39 (67.2%) of those residents completed the assessment in 2 consecutive years. Based on data from all residents' first assessment, statistically significant differences (p Conclusions The Omni is a novel composite assessment tool for surgical technical skill that utilizes objective measures and scoring algorithms to evaluate performance. In this pilot study, 3 tasks demonstrated discriminative ability of performance by CY, and all 5 tasks demonstrated construct validity by showing longitudinal improvement in performance. Additionally, the Omni has adequate internal consistency for a formative assessment. These results suggest the Omni holds promise for the evaluation of resident technical skill and early identification of outliers requiring intervention.