Measuring the Benefits of Examinee‐Selected Questions

Allowing students to choose the question(s) that they will answer from among several possible alternatives is often viewed as a mechanism for increasing fairness in certain types of assessments. The fairness of optional topic choice is not a universally accepted fact, however, and various studies have been done to assess this question. We examine an important class of experiments that we call C1-A, “choose one, answer all,” designs, and point out an important problem that they face. We suggest two analytical methods that can be used to circumvent this problem. We illustrate our ideas using the data from Bridgeman et al. (1997). Our reanalysis of these data show: (a) that differential topic difficulty exists in real choice data, (b) that it affects naive analyses of such data and masks the effects, positive or negative, of examinee choice, (c) that in this study there is a measurable and positive effect of examinee choice that follows predicted patterns in most but not all cases, (d) that the beneficial strength of examinee choice varies from case to case, and (e) that while the benefits of choice in terms of average points scored on the essays are usually positive, there is a substantial amount of variation around these averages and it is not uncommon for “incorrect” choices to be associated with higher test performance.