Workplace-based assessments in postgraduate medical education - a hermeneutic review.

PURPOSE: Since their introduction, workplace-based assessments (WBAs) have proliferated throughout postgraduate medical education. Previous reviews have identified mixed findings regarding WBAs' effectiveness, but have not considered the importance of user-tool-context interactions. The present review was conducted to address this gap by generating a thematic overview of factors important to the acceptability, effectiveness and utility of WBAs in postgraduate medical education. METHOD: This review utilised a hermeneutic cycle for analysis of the literature. Four databases were searched to identify articles pertaining to WBAs in postgraduate medical education from the UK, Canada, Australia, New Zealand, the Netherlands and Scandinavian countries. Over the course of three rounds, 30 published articles were thematically analysed in an iterative fashion to deeply engage with the literature to answer three scoping questions concerning acceptability; effectiveness; and assessment training. As each round was coded, themes were refined and questions added until saturation was reached. RESULTS: Stakeholders value WBAs for permitting assessment of trainees' performance in an authentic context. Negative perceptions of WBAs stem from misuse due to low assessment literacy; disagreement with definitions and frameworks; and inadequate summative use of WBAs. Effectiveness is influenced by user (e.g. engagement, assessment literacy) and tool attributes (e.g. definitions, scales), but most fundamentally by user-tool-context interactions, particularly trainee-assessor relationships. Assessor assessment literacy must be combined with cultural and administrative factors in organisations and the broader medical discipline. CONCLUSIONS: The pivotal determinants of WBAs' effectiveness and utility are the user-tool-context interactions. From the identified themes, we present 12 lessons learnt regarding users, tools and contexts to maximise WBA utility, including the separation of formative and summative WBA assessors, use of maximally useful scales, and instituting measures to reduce competitive demands.