Examination of the Inter-Rater Agreement among Faculty Marking a Research Proposal on an Undergraduate Health Course

Purpose: This research looked at inter-rater agreement among faculty marking a researc course. The purpose of this study was to investigate if there were significant differences between marks as well as the comme given on research proposals that were double marked by two faculty members, where faculty marks and comments. Subjects: The subjects for this study consisted of 80 final year health students who submitted a research proposal and 13 full-time and 10 part-time faculty members who double marked the proposals. Meth the proposals independently and were required to meet with each other and discuss the marks given, reasons for the mark and the comments. Markers were asked to come to an agreement about the mark and the comments. Analyses: Agreem was assessed using a Bland- Altman plot. Weighted Cohen's Kappa was used to estimate the agreement between the classifications given by markers. Comments were analyzed for differences using thematic analysis. Results: There was a wide discrepancy in the classification of students between markers. The weighted proportions found to agree on classificatio n was 46%. Analysis of the comments indicated a wide discrepancy between markers. Conclusion: The outcomes of this study are similar to previous studies that have looked at inter written assignments. Further exploration of the inter results in a transparent system is needed.