Written-expression curriculum-based measurement (WE-CBM) is used for screening and progress monitoring students with or at risk of learning disabilities (LD) for academic supports; however, WE-CBM has limitations in technical adequacy, construct representation, and scoring feasibility as grade-level increases. The purpose of this study was to examine the structural and external validity of automated text evaluation with Coh-Metrix versus traditional WE-CBM scoring for narrative writing samples (7-min duration) collected in fall and winter from 144 second- through fifth-grade students. Seven algorithms were applied to train models of Coh-Metrix and traditional WE-CBM scores to predict holistic quality of the writing samples as evidence of structural validity; then, external validity was evaluated via correlations with rated quality on other writing samples. Key findings were that (a) structural validity coefficients were higher for Coh-Metrix compared with traditional WE-CBM but similar in the external validity analyses, (b) external validity coefficients were higher than reported in prior WE-CBM studies with holistic or analytic ratings as a criterion measure, and (c) there were few differences in performance across the predictive algorithms. Overall, the results highlight the potential use of automated text evaluation for WE-CBM scoring. Implications for screening and progress monitoring are discussed.
Many students struggle with the basic skill of writing, yet schools lack technically adequate screening measures to identify students at risk in this area. Measures that allow for valid screening decisions that identify students in need of interventions to improve performance are greatly needed. The purpose of this study was to evaluate the validity and diagnostic accuracy of early writing screeners. Two early writing screening measures, Picture Word and Word Dictation, were administered to a diverse sample of 95 kindergarten students, almost half of whom were classified as English language learners and almost 70% identified ethnically as Hispanic. It was hypothesized that the early writing screening measures would demonstrate moderate to strong relationships with a standardized norm-referenced measure of written expression and adequate diagnostic accuracy for identifying kindergarten students at risk. Findings indicate that concurrent validity coefficients for both Picture Word and Word Dictation tasks ranged from .32 to .70 with the Written Expression cluster of the Woodcock–Johnson Tests of Achievement–IV and .26 to .61 with the Writing Samples and Sentence Writing Fluency subtests. Diagnostic accuracy results suggest these measures are a promising option for screening early writing skills. Implications for practice and directions for future research are discussed.
