Support vector machines for the automatic grading of assignments

Support vector machines (SVMs), a new artificial intelligence technique, are applied to the automatic grading of English-language assignments, a multiple classification problem. Thirty-two essays from a third-year course in the Bachelor of Information Technology program at Sohar University are utilised. All assignments are from the "Information Science" course in Semester 1 2003/2004 and have been written in English by native speakers of Arabic. As part of pre-processing, Figures were removed to allow for a text-only assessment of the essays by a machine learning system. The original assignments were given marks in the range of 0-40. Sohar University uses the Australian grading system (grades from 1 to 7) and for the purpose of this study, marks were mapped to grades. These grades are the targets for supervised machine learning by use of support vector machines and as such the classification problem here is more course-grained than the human marking of the essays. For evaluation purposes, the multiple classification problem has been mapped to several binary decision tasks, e.g. "high distinction vs. else", "high distinction and distinction vs. else" and so on. While there was limited learning for the "high distinction vs. else" case (possibly due to the unbalanced data set), results improved if grades were combined, e.g. "high distinction and distinction vs. else". Leave-one-out crossvalidation indicates that accuracy, precision and recall is well above 80% and pass/failure is predicted with high accuracy.