Automated AUT scoring using a Big Data variant of the Consensual Assessment Technique : Final Technical Report

In this final technical report, we present the results of our Abbas Foundation Test Development Funds project “The development of a valid and usable creativity test requires big-data and psychometrics”. The project consisted of two phases: (1) creating a large database of responses for the Dutch version of the “Alternative Uses Task” (AUT) and (2)developing an automated scoring algorithm based on the Consensual Assessment Technique and testing its reliability and validity. The main aim of the project was to establish a reliable and automated way to score the AUT that will make future data coding faster and more cost-efficient. Meanwhile, the problem of sample-specific scoring would be solved because automated scoring guarantees consistency, i.e. that the same AUT response receives the same creativity score regardless of where the data was collected and scored. We developed an algorithm that essentially scores AUT responses using the Consensual Assessment Technique based on expert ratings of similar responses from our database of over 70,500 AUT responses. Based on two validation studies, the results show that our algorithm was the best ‘rater’ and reliably scores new AUT responses similarly to experts. Furthermore, the test-retest and alternate form reliability as well as convergent, discriminant and predicted validity of automated scoring is on par with that of expert scoring. There is still room for improvement, but the current version of our AUT scoring algorithm is a reliable alternative to the time-intensive and costly expert scoring methods.