Development and validation of a portfolio assessment system for medical schools in Korea

Purpose Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and then to assess its validity and reliability. Methods We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment with an expert group comprising 2 medical education specialists, 2 professors involved with education at the College of Medicine, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the "Self-Development and Portfolio II" course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020. Results A portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the 1st round of evaluation grades submitted, all assessment areas except ''goal-setting" showed a high ICC of 0.81 or higher. After the 1st round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher. Conclusion We confirmed that if an assessor with appropriate training conducts a portfolio assessment based on specified standards through a systematic procedure, the results are reliable.