Evaluating the effectiveness of a police department’s early intervention system

Police departments around the country are implementing Early Intervention Systems (EIS) to identify officers who may be exhibiting problematic or unprofessional behaviors. The goal of EIS is to minimize officer misconduct and increase officer accountability. To evaluate whether EIS can actually differentiate “problem” from “non-problem” officers, we analyzed the performance of officers from incident reports of police–citizen interactions. Using a blind scoring method, we evaluated performance from 1000 police reports; 500 randomly selected reports from EIS-flagged officers (treatment group) and 500 randomly selected reports from non-flagged officers (control group). Six hundred and sixty-seven reports contained relevant performance data. The interval-level metrics used to score officer performance were developed by Vila and colleagues (2016, 2018) to assess performance—expressed as a percentage—across a range of police–citizen encounters. The overall performance score assigned to officers across all 667 incident reports was 80.46% (SD = 8.75%). When separated into EIS-flagged and non-EIS–flagged incidents, performance scores were 80.63% (SD = 8.58%) compared to 80.27% (SD = 8.95). There was not a statistically significant difference between EIS-flagged and non-EIS–flagged performance. The EIS evaluated does not appear to be differentiating between problem behavior and non-problem behavior. This suggests that the “thresholds” used to identify problem officers are not working effectively.