Interrater reliability issues in multicenter trials, part I: Theoretical concepts and operational procedures used in Department of Veterans Affairs Cooperative Study 394

This article describes a standardized method for establish­ ing and maintaining desired levels of interrater reliability (IRR) in multicenter trials. The procedure involves six steps: distribution of procedural guides, distribution of an introduc­ tion tape, initial distribution of patient interviews to rate, training at the study kickoff meeting, ongoing IRR monitor­ ing, and group training throughout the study. This method is being used In a national Veterans Affairs Cooperative Study (CS #394), involving nine sites to examine the treatment effects of vitamin E on tardive dyskinesia. The six-step stan­ dardized process allowed for early detection of areas of con­ cern in assessment administration. When comparing intra­ class correlation coefficients (ICCs) at different points in the initial training, the Barnes Akathisia Scale and Anchored Brief Psychiatric Rating Scale reliability improved from 0.68 to 0.74 and from 0.54 to 0.87, respectively. After analyzing the ratings collected prior to the start of CS #394, data were collected to conduct the first check on Abnormal Involuntary Movement Scale (AIMS) IRR during enrollment; the esti­ mated ICC for the AIMS had decreased from 0.87 to 0.60. Raters were instructed to re-assess the subjects from the first videotape on the AIMS and received additional training. The re-rating indicated very good reliability, 0.84. IRR was measured once for the Global Assessment of Functioning Scale resulting in an ICC of 0.90. The companion article (Part Psychopharmacology Bulletin 33(1):53-57,1997.