Using Pairwise Comparisons in the Online Social Moderation of Performance Assessment

Assessing the performance of a student involves some form of judgement, and where more than one assessor is involved this usually requires some form of moderation to ensure consistent and fair results. Often this involves meetings or communication between assessors, which is referred to as social moderation. This paper reports on a study that investigated the use of online technologies to support a form of social moderation of artworks submitted for assessment in a senior secondary school course in Western Australia. Online systems were used to facilitate communications and provide access to digital representations of the submissions along with assessment tools. In particular a pairwise comparison judging online tool was used. This approach to social moderation was tested in a realistic context involving a sample of 12 teachers from rural schools for whom face-to-face meetings would be difficult. The aim was to investigate whether the use of these online systems would support good moderation outcomes and valuable professional learning for those involved. The study found that this approach to online social moderation was feasible, and participants perceived that it had improved the consistency of their judgements because they had developed an improved understanding of the assessment criteria and standard of work. However, analysis of scores and reliability data suggested some were not adequately consistent, and it was likely that this was due to their inexperience in assessing such work. Therefore some changes to the processes of this form of online social moderation were recommended.