No Pain, No Gain? The Importance of Measuring Course Workload in Student Ratings of Instruction

Samples of about 200 undergraduate courses were investigated in each of 3 consecutive academic terms. Course survey forms assessed evaluative ratings, expected grades, and course workloads. A covariance structure model was developed in exploratory fashion for the 1st term's data, and then successfully cross-validated in each of the next 2 terms. The 2 major features of the successful model were that (a) courses that gave higher grades were better liked (a positive path from expected grades to evaluative ratings), and (b) courses that gave higher grades had lighter workloads (a negative relation between expected grades and workload). These findings support the conclusion that instructors' grading leniency influences ratings. This effect of grading leniency also importantly qualifies the standard interpretation that student ratings are relatively pure indicators of instructional quality. Student ratings have been both praised as being valid and efficient and criticized as being insensitive and misleading.' The present research proceeds from an intermediate viewthat student ratings may be imperfect but are nevertheless useful and are also improvable through research. The specific aim of the present research was to construct and confirm a covariance structure model that could identify sources of desired or undesired influences on student ratings. The most ambitious previous effort to describe and confirm a covariance structure model of student ratings has been Marsh's (1991) hierarchical confirmatory factor analysis of &ta provided by 35 items of the SEEQ (Student Evaluations of Educational Quality) inventory. Marsh reported substantial confirmatory support for a nine-factor first-order structure overlaid with a four-factor structure in which the higher order factors represented similarity relations among the nine first-order factors. In contrast with Marsh's (1991) aim of analyzing the dimensional structure of student ratings, the present research sought to evaluate theories of causal influences operating on student ratings. Alternative theories of the influences that affect student ratings imply different patterns of relationships among three categories of measures: (a) evaluative ratings, (b) expected grades, and (c) course workloads.