Assessing Some of the Properties of Longer Blocks in the 1992 NAEP Reading Assessment.

At Grades 8 and 12, the 1992 NAEP reading assessment contained a small number of 50 minute blocks, in addition to the usual 25 minute blocks. In order to determine whether to incorporate the 50 minute blocks into the operational scaling, this study sought to determine whether longer blocks measure a different construct from that assessed by the 25 minute blocks. Structural equation modeling tested the hypothesis that the structural parameters relating reading ability to demographic variables do not differ across block type. A multiple group analysis, where type of block (25 minute or 50 minute) defined the two groups, was used. The null hypothesis for this comparison was that the two types of blocks measure the same trait but may differ in observed mean and variance. Results of the main analysis did not reject the hypothesis of invariant structural parameters, and so the 50 minute blocks were incorporated in the 1992 NAEP scales. Sensitivity analyses indicated that this conclusion was moderately robust to assumptions made about missing data for items which were not reached, although Grade 8 results were more robust than those for Grade 12. Analyses using other measures of fit yielded the same pattern of results, although the magnitude of the χ2 statistics were affected by the fit measure chosen, particularly the asymptotically distribution free method. Attempts to replicate the main analysis in independent samples yielded similar χ2 values at Grade 8, but Grade 12 yielded χ2 values which were substantially higher for some of the samples. The Grade 12 results raise questions as to the generalizability of the main analysis. Alternatives to the reliance on χ2 measures are discussed for future research.