Use of Item Response Theory to Facilitate Concept Inventory Development

This paper explores ways that item response theory techniques can be used to gain additional insight into question behavior and student misconceptions, in addition to being valuable tools for development and evaluation practices. Therefore, this paper presents how IRT analysis has been used in the development of the Statistics Concept Inventory to make decisions during the revision process, enabling sample independent comparisons of question versions to be made as well as giving insight as to how a question behaves over a range of abilities. Reliability of the instrument will be assessed from a classical test theory perspective and an IRT perspective. Coefficient alpha (a sample dependent measure) has varied from semester to semester, but has generally been near 0.75. The item response theory reliability estimate obtained from all the data (based on the concept of test information) was found to be 0.78. This initial work is now being extended through the development of "clicker" or personal response system questions. These questions are again driven by student misconceptions where each distractor has a meaning to the instructor and is able to alert him/her to specific and/or common misconceptions within their class.