畢業後一般醫學訓練住院醫師「客觀結構式臨床技能測驗」的可行性評估

Background The objective structured clinical examination (OSCE) is known as a reliable and valid assessment of both undergraduate medical students and in several post-graduate medical specialties. However, there are concerns regarding its feasibility and whether it is suitable as an assessment of competence based medical education (CBME). There is limited data on the use of multi-specialty OSCE for assessment in postgraduate medicine. Taiwan also has an unique postgraduate year (PGY) general medicine training program and there is limited reliability or validity evidence of OSCE in this population. Aim and Research Questions 1. Evaluate the feasibility of multi-specialty OSCE in the post-graduate year (PGY) general medicine training program 2. Examine the reliability and validity of multi-specialty OSCE in this population and setting 3. Identify potential factors that affect reliability of this assessment Concise Methods During June 2016 and June 2017, 83 PGY residents participated in four seperate end-of-year OSCE assessment at Far Eastern Memorial Hospital (FEMH). The design, blueprint, faculty and standardized patient training, outcomes, as well as questionnaire responses were reported as evidence of technical feasibility. Economically feasibility was evaluated through estimating the cost of applying an OSCE exam. Reliability and validity evidence was gathered through analysis of across-station, across-item, inter-rater reliability, as well as scores and correlations parameters. The effects of rater training and different station lengths and having multiple raters were examined. Results OSCE was a feasible but resource demanding method of assessment with positive response and satisfaction from faculty and trainees. The across-station station reliability (Cronbach’s α 0.104~0.464) and across-item reliability (Cronbach’s α -0.217 to 0.483) were low. Overall good correlation between checklist items with global rating (Coefficient of determination of R2 0.32~0.907) and moderate to good inter-rater reliability was found. Using the scores of multiple raters improved across-station reliability and inter-rater of three raters was higher than two raters. Discussion Feasibility of multi-specialty OSCE in PGY residents of the general medicine training program in one hospital was provided. However, feasibility using the same format may not apply to all hospitals, depending on the size of each program. It was technically and economically feasible at FEMH due to the strong support of the hospital’s medical education department and faculty and also through the close collaboration between the different specialties involved. Across-station and across-item reliability were in general low, and a potential explanation is evaluation of distinct constructs due to the multi-specialty design and measure of multiple clinical skills and competence in long stations. We found that having multiple raters improved across-station reliability. Based on previous literature, other potential factors that may affect across-station and across-item reliability include number of stations, length of checklist and clinical skill tested. Overall inter-rater reliability was good and three raters compared to two raters in general improved inter-rater reliability. Overall correlation between checklist and global rating, inter-rater reliability, validity in the form of content, response process, internal structure and criterion validity provided evidence towards overall fair reliability and validity of this assessment. Conclusion A multi-specialty OSCE as an end-of-year summative and formative assessment in Taiwanese PGY residents general medicine training program is technically and also economically feasible. It can be used to assess multiple core competencies and important clinical skills. Reliability may be affected by various factors, and the use of double raters is an effective way to increase reliability.