ORIGINAL ARTICLE The Americleft Speech Project: A Training and Reliability Study

Kathy L. Chapman, Ph.D., Adriane Baylis, Ph.D., Judith Trost-Cardamone, Ph.D., Kelly Nett Cordero, Ph.D., AngelaDixon, M.A., Cindy Dobbelsteyn, M.Sc., Anna Thurmes, M.A., Kristina Wilson, Ph.D., Anne Harding-Bell, Ph.D.,Triona Sweeney, Ph.D., Gregory Stoddard, Ph.D., Debbie Sell, Ph.D.Objective: To describe the results of two reliability studies and to assess the effect of trainingon interrater reliability scores.Design: The first study (1) examined interrater and intrarater reliability scores (weighted andunweighted kappas) and (2) compared interrater reliability scores before and after training on theuse of the Cleft Audit Protocol for Speech–Augmented with British English-speaking children.The second study examined interrater and intrarater reliability on a modified version of the CleftAudit Protocol for Speech–Augmented (Cleft Audit Protocol for Speech–Augmented–AmericleftModification) with American and Canadian English-speaking children. Finally, comparisons weremade between the interrater and intrarater reliability scores obtained for Study 1 and Study 2.Participants: The participants were speech-language pathologists from the AmericleftSpeech Project.Results: In Study 1, interrater reliability scores improved for 6 of the 13 parameters followingtraining on the Cleft Audit Protocol for Speech–Augmented protocol. Comparison of thereliability results for the two studies indicated lower scores for Study 2 compared with Study 1.However, this appeared to be an artifact of the kappa statistic that occurred due to insufficientvariability in the reliability samples for Study 2. When percent agreement scores were alsocalculated, the ratings appeared similar across Study 1 and Study 2.Conclusion: The findings of this study suggested that improvements in interrater reliabilitycould be obtained following a program of systematic training. However, improvements were notuniform across all parameters. Acceptable levels of reliability were achieved for thoseparameters most important for evaluation of velopharyngeal function.KEY WORDS: Americleft, cleft palate, listener agreement, listener reliability, perceptual ratings,speech outcome, training