Systematic Review: Online Crowdsourcing to Assess Perceptual Speech Outcomes

Abstract Background Speech is integral for human interaction and development. Speech assessments are critical in the growing child, especially in the surgical evaluation of patients undergoing cleft palate and speech surgeries. Online crowdsourcing enables layperson raters, allowing rapid and large-scale data collection. This systematic review analyzes the utility of online crowdsourcing to evaluate perceptual speech outcomes. Methods Terms related to “crowdsourcing” and “speech” were searched on PubMed, Scopus, CINAHL, Cochrane CENTRAL, and PsycINFO on August 16, 2017, returning 2812 unique articles. Inclusion and exclusion criteria concentrated on online crowdsourcing of perceptual speech outcomes: titles led to 140 abstracts that yielded 35 full-text articles, of which eight articles met criteria for analysis. Results All studies used Amazon Mechanical Turk for online crowd raters, and one used an additional crowdsourcing site (CrowdFlower). Disordered speech was provided by 376 speakers, for which 2203 crowd workers produced over 700,000 unique ratings. Five studies compared crowdsourced assessments to gold standards and found high concordances. Data collection time ranged from 59 min to 23 h, with worker payments ranging from $0.05 to $2.00 per task. Studies examined child pronunciation of the /r/ sound, dysarthria in Parkinson's speech, and articulation of English words produced by non-English speakers learning English. Conclusions Online crowdsourcing for perceptual speech outcomes provides high-quality data consistent with previous speech-assessment standards in a rapid, cost-effective manner. This novel methodology incorporates lay perspective of speech intelligibility and has the potential to revolutionize surgical speech outcome assessments, including cleft palate and speech surgery.