A comparison of phone-based and onsite-based fidelity for Assertive Community Treatment in Indiana

Poor implementation of evidence based practices (1, 2) and its toll on program outcomes are a critical concern for mental health services (3, 4). One accepted strategy to improve implementation is to verify program fidelity (5, 6). However, with the increasing number of evidence based practices, the need to conduct fidelity measurement has begun to place a very high burden on agencies charged with ensuring service quality. For example, the current standard fidelity instrument for ACT, the Dartmouth Assertive Community Treatment Scale (DACTS; 7), requires one day for the onsite visit and another day to score and write the report for quality improvement feedback. In response to these and related problems, a 2007 national task force met to identify alternative approaches for ensuring quality (8). Among the strategies discussed was alternative fidelity methods such as phone-administered assessments. Although phone-administered fidelity has been used successfully in predicting outcomes (9), no research has validated phone-administered assessment compared to onsite assessment. We examined the interrater reliability and concurrent validity of phone-administered fidelity as applied to Assertive Community Treatment. A secondary question asked whether validity and reliability are higher when using raters with prior experience with fidelity assessment or experience with the site.