Audience Response System-Based Evaluation of Intelligibility of Children's Connected Speech - Validity, Reliability and Listener Differences.

Abstract Purpose We assessed audience response systems (ARS)-based evaluation of intelligibility, with a view to find a valid and reliable intelligibility measure that is accessible to non-trained participants. In addition, we investigated potential listener differences between pediatric speech and language pathologists (SLPs) and untrained adults. Method Sixteen one-minute samples of connected speech were compiled, collected from 14 children with a speech sound disorder (SSD) and two children with typical speech. 16 SLPs and 13 untrained adults participated in a series of ARS listening sessions, where they were fitted with headphones and hand controls, and instructed to click a button whenever they did not understand the child speaking. Listeners’ button clicks were registered and, for each speech sample, totaled into an (un)intelligibility index. The proportion of syllables perceived correctly – based on orthographic listener transcripts – was used as a reference score of intelligibility. Results The ARS-based intelligibility scores correlated strongly with the intelligibility reference score. Reliability was high across listener groups and weaker for single listeners. No significant difference was found between the evaluations of SLPs and untrained adults. Conclusions ARS-based evaluation offers a valid and reliable measure of intelligibility of particular value in research as a practical tool for collecting input from listeners without experience or knowledge of SSDs. We stress that the ARS design presupposes a listener panel, and that evaluations obtained from individual listeners are predictably inadequate in terms of reliability.