Brainwaves in your pager

techniques, with the goal of reaching consensus in the application of the criteria of the assessment tool. At these meetings, site directors share their own preferences and biases in grading, and are thus able to identify idiosyncratic scoring tendencies and the types of errors they tend to weight disproportionately to the other graders. These discussions are usually successful in reconciling disagreed-upon scoring items. Evaluation of results and impact To assess inter-rater reliability, a total of 52 write-ups were randomly selected from between the two sites during a single rotation. Each write-up was independently reviewed by all three site directors. Each site director was blinded to the identity of the author as well as to the site from which the write-up was drawn. Reliability analysis using average measures intra-class correlation coefficients showed excellent inter-rater reliability for the total scores (0.88). Reliability varied within the individual components of the evaluation and was highest for the Discussion and Physical Examination sections (0.81 and 0.84, respectively). The Problem List, Complexity and History of Present Illness components had lower levels of reliability (ranging from 0.57 to 0.61). There has been a noticeable reduction in student complaints about idiosyncratic scoring. Because of the perceived success of this new method, we have now also consolidated observed physical examination assessments into the hands of the site directors and we plan to measure the reliability of this technique.