Are intermediate constraint question formats useful for evaluating student thinking and promoting learning in formative assessments

Abstract There is a broad movement in science classes to emphasize higher-order thinking and complex skills over simple accumulation of facts. A major barrier to making this shift is a lack of good assessment tools for evaluating higher-order thinking that can be used with low effort by instructors. Computerized assessments aid instructors and increase efficiency, and potentially allow for immediate feedback. But easily-used question formats tend to be highly constrained (e.g. multiple-choice) and thus less informative on student thinking, while low constraint open-response formats (e.g. essay questions) are difficult to grade automatically. This is particularly problematic for formative assessments where immediate feedback specific to a student's current understanding is important to their learning. Here we explore two question formats that are intermediate in degree of constraint between open-response and multiple-choice formats. Intermediate constraint formats have the potential to allow some response construction while still enabling automatic feedback. We examine 15 questions in the two formats, with data from student interviews as well as from use in dozens of classrooms (both high school and university) as part of their regular curriculum. We find that the intermediate constraint questions enable automatic scoring of answers into specific categories of understanding and confusion, thus allowing for specific feedback to each student; that the intermediate constraint formats do encourage students to construct their own answers to a greater degree than multiple-choice; that they are usable by students; and that questions in those formats can show similar ranges of student answers as open-response. Further, using a split-class design in high school classrooms, we show that intermediate constraint questions can impact student learning to a greater degree than short answer questions. We conclude that these two question formats, and by extension perhaps other question formats with intermediate levels of constraint, are a useful addition to the toolbox of the educational software developer, but because they require a higher effort to construct than other formats, they should be reserved for key points in an online lesson.