An investigation of the reliability of using comparative judgment to score creative products

ABSTRACTIn this article we describe a three-year study that was conducted in three phases to evaluate the feasibility of assessing digitized portfolios of student creative work for high-stakes purposes. The first two phases suggested that creative work could be digitized with adequate fidelity, and that students could submit their own work from schools to an online portfolio system. These portfolios were assessed using online tools to facilitate both an analytical and comparative judgment method of scoring, with the latter appearing to be the more reliable. In the final phase this approach was evaluated to facilitate moderation and professional learning of standards with rural teachers. The study showed that comparative judgment is a viable and reliable method for assessing digitized creative products and coupled with online communication systems provides an effective and efficient approach to scoring, moderation and teacher professional development, particularly in rural locations.