Assessment of Interrater Reliability of the Emergency Severity Index After Implementation in Emergency Departments in Jamaica Using a Learning Collaborative Approach

Abstract Introduction In 2016, the Ministry of Health in Jamaica selected the Emergency Severity Index as the triage tool to be used nationally. This study evaluated the effectiveness of this approach by assessing the interrater reliability among new users trained with minimal resources by 2 experienced trainers, 1 local and 1 international. Methods A retrospective case series review was conducted within an online learning collaborative framework. After completion of the training, the participants from each of the 19 clinical sites were asked to submit 2 triage cases per month for blinded review by the expert trainers. The triage categories assigned by each reviewer were compared with those assigned by the newly trained Emergency Severity Index providers. A weighted kappa value was calculated to assess the degree of agreement between the sites and the expert trainers. Results A total of 166 cases were received over the study period. Participation in the learning collaborative was consistently below 50%. The interrater reliability between the expert trainers (κ = 0.48) as well as between each scorer and each accident and emergency department site (κSF = 0.33, κPT = 0.26) was low, although there was improvement over the study period. Incomplete triage documentation limited raters' ability to assign triage categories and assess interrater reliability. Discussion Despite a rigorous implementation process, the interrater reliability of the Emergency Severity Index skills of Jamaican emergency nurses and doctors when compared with that of the 2 experts was poor. Several areas were identified for strengthening. Considerations for the implementation of the Emergency Severity Index in countries outside of the US were also discussed.