To the Editor: Retrospective “Ranking” of Former Graduate Trainees by Senior Faculty Who Worked With Them During Their Residency
0
Citation
3
Reference
10
Related Paper
Abstract:
We read with interest the article, “A Critical Disconnect: Residency Selection Factors Lack Correlation With Intern Performance,” by Burkhardt et al and would like to commend the authors for their valuable investigation into the status quo of the residency match process.1 Their conclusion corroborates our experience. All residency programs seek to identify and recruit whom they believe are the most promising medical school graduates—typically based on USMLE scores, class ranking, clinical grades, letters of recommendation, etc.2 The assumption in this ranking process is that competitiveness via these criteria is the best available metric for judging future performance of resident physicians. Existing evidence suggests that programs are not particularly efficient at determining whether applicants selected in this manner will become top performers during their residency training.3We suspected that current applicant metrics are not adequate predictors and ultimately do not correlate strongly with eventual resident performance. We examined National Resident Matching Program (NRMP) rank order list data over a 15-year period of anesthesiology residents from our institution, with class sizes varying from 6 to 12 residents per year. We were able to ask 4 longstanding, full-time faculty members who were present for the entire residency experience of these classes to reflect and “re-rank” the matched candidates relative to each other. We were limited to faculty who felt that they remembered the candidates well enough to complete the task. Each eligible faculty member independently ranked the past graduates of each class from highest to lowest compared to their class peers. The ranking was based on the faculty's recollection of the residents after working with them during their residency. Faculty essentially re-ranked these residents based on their overall impression of the graduates after they experienced their individual strengths, shortcomings, and limitations. Descriptive analysis was performed comparing correlation between the percentile in the actual NRMP rank list position vs faculty raters' ranking of each resident.From our preliminary findings, there was general agreement among the faculty raters with considerable correlation (R = 0.567). While some variation in rankings among the faculty existed, no faculty member was an obvious outlier. Despite some evidence that residents taken higher in the NRMP rank list received higher impression ratings from the faculty, the association was quite weak with a lower correlation (R = 0.204).Our findings reinforce the authors' results and suggest some insight into how difficult it is to identify and predict potential for clinical excellence. Extensive time and resources are devoted to parameters that, in our opinion, weakly predicts future resident performance. Exploration of nontraditional metrics may prove more valuable than currently emphasized parameters.Keywords:
Graduate medical education
Rank (graph theory)
The transition from American Osteopathic Association (AOA) and Accreditation Council for Graduate Medical Education (ACGME) residency matches to a single graduate medical education accreditation system culminated in a single match in 2020. Without AOA-accredited residency programs, which were open only to osteopathic medical (DO) graduates, it is not clear how desirable DO candidates will be in the unified match. To avoid increased costs and inefficiencies from overapplying to programs, DO applicants could benefit from knowing which specialties and ACGME-accredited programs have historically trained DO graduates.This study explores the characteristics of residency programs that report accepting DO students.Data from the American Medical Association's Fellowship and Residency Electronic Interactive Database Access were analyzed for percentage of DO residents in each program. Descriptive statistics and a logit link generalized linear model for a gamma distribution were performed.Characteristics associated with graduate medical education programs that reported a lower percentage of DO graduates as residents were surgical subspecialties, longer training, and higher US Medical Licensing Examination Step 1 scores of their residents compared with specialty average. Characteristics associated with a higher percentage of DO graduates included interviewing more candidates for first-year positions and reporting a higher percentage of female residents.Wide variation exists in the percentage of DO graduates accepted as residents among specialties and programs. This study provides valuable information about the single Match for DO graduates and their advisers and outlines education opportunities for the osteopathic profession among the specialties with low percentages of DO students as residents.
Graduate medical education
Cite
Citations (18)
To determine whether the 2003 Accreditation Council for Graduate Medical Education (ACGME) duty hours reform affected medical knowledge as reflected by written board scores for internal medicine (IM) residents.The authors conducted a retrospective cohort analysis of postgraduate year 1 (PGY-1) Internal Medicine residents who started training before and after the 2003 duty hour reform using a merged data set of American Board of Internal Medicine (ABIM) Board examination and the National Board of Medical Examiners (NMBE) United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge test scores. Specifically, using four regression models, the authors compared IM residents beginning PGY-1 training in 2000 and completing training unexposed to the 2003 duty hours reform (PGY-1 2000 cohort, n = 5,475) to PGY-1 cohorts starting in 2001 through 2005 (n = 28,008), all with some exposure to the reform.The mean ABIM board score for the unexposed PGY-1 2000 cohort (n = 5,475) was 491, SD = 85. Adjusting for demographics, program, and USMLE Step 2 exam score, the mean differences (95% CI) in ABIM board scores between the PGY-1 2001, 2002, 2003, 2004 and 2005 cohorts minus the PGY-1 2000 cohort were -5.43 (-7.63, -3.23), -3.44 (-5.65, -1.24), 2.58 (0.36, 4.79), 11.10 (8.88, 13.33) and 11.28 (8.98, 13.58) points respectively. None of these differences exceeded one-fifth of an SD in ABIM board scores.The duty hours reforms of 2003 did not meaningfully affect medical knowledge as measured by scores on the ABIM board examinations.
Graduate medical education
Board certification
Demographics
Educational measurement
Institutional review board
Cite
Citations (17)
Purpose To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. Method The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013–2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME’s core competency domains and overall. Results SI scores predicted midyear and year-end overall performance ( r = 0.18 and 0.19, respectively, P < .05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies ( r = 0.23, r = 0.22, and r = 0.20, respectively, P < .05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care ( ΔR = 0.05), interpersonal and communication skills ( ΔR = 0.09), and professionalism ( ΔR = 0.09; all P < .05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. Conclusions SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.
Graduate medical education
Milestone
Licensure
Educational measurement
Core competency
Cite
Citations (39)
Milestone
Graduate medical education
Categorical variable
Cite
Citations (0)
The Accreditation Council for Graduate Medical Education (ACGME) introduced duty hour reforms for resident physicians in 2003, which included the current 80-hour maximum work week, averaged over 4 weeks; subsequent 2011 work hour reforms included policies limiting maximum shift lengths for intern physicians.1 Both reforms stemmed from a growing concern about the effects of resident fatigue on both patient safety and resident well-being.1,2 There has been much debate surrounding these reforms, and opponents have argued that they have negatively affected resident education.2,3 After assessing these concerns and the findings from a recent study of flexible models of resident education, the ACGME approved new 2017 work hour standards.2,4However, there remains a lack of data on the actual impact of work hour reforms. Previous studies have evaluated surgical resident certification examination scores as a proxy for quality of resident education in the context of duty hour reforms.3 We analyzed first-time taker pass rates for the American Board of Internal Medicine (ABIM) initial certification examination, a standardized examination independent from the ACGME, to evaluate whether 2003 or 2011 ACGME duty hour reforms were associated with differences in internal medicine (IM) resident pass rates on a national certification examination.5Our analysis reflected a 20-year span from 1996 to 2016 and included 151 844 test takers. Since reforms were implemented in 2003 and 2011, aggregate scores for the 3 surrounding periods (1996–2003, 2004–2011, and 2012–2016) were compared with an analysis of variance (ANOVA). All analyses were conducted using SAS version 9.4 (SAS Institute Inc, Cary, NC). Use, reproduction, and analysis of these data were approved by the ABIM. This work was reviewed by the Institutional Review Board of the Massachusetts General Hospital and deemed exempt.The number of first-time takers ranged from 6751 (2003) to 7853 (2016); pass rates ranged from 82% (1996) to 94% (2007; figure). Mean pass rates ± SD were 86% ± 2.9 (1996–2003), 90% ± 3.3 (2004–2011), and 87% ± 2.1 (2012–2016). There were no significant differences in pass rates for the 3 periods (ANOVA, P > .05).Our analysis did not reveal any significant difference in examination pass rates for IM residents surrounding either change in the ACGME work hour standards. That finding does not support previously held concerns that work hour reforms have negatively impacted resident education, at least as measured by a standardized test of medical knowledge—a concern that, in part, led to the 2017 revisions in the ACGME duty hour standards.2–4Our work has limitations. This was an observational analysis that did not account for possible unobserved confounders. Fortunately, the initial certification examination is regularly reviewed by the ABIM to equate scores among administrations; thus, the nonsignificant year-to-year variation seen in examination scores likely represents natural variation due to varying ability of the examination taker populations.Given the ongoing debate around duty hour reforms, the recent ACGME policy revisions, and the ongoing efforts by researchers conducting the iCOMPARE trial (an IM study piloting alternative models of resident education), further study is warranted to determine whether any other educational outcomes have been measurably affected by duty hour reforms.5
Graduate medical education
Proxy (statistics)
Cite
Citations (5)
Recently, the Institute of Medicine examined resident duty hours and their impact on patient safety. Experts have suggested that reducing resident work hours to 56 hours per week would further decrease medical errors. Although some reports have indicated that cutbacks in resident duty hours reduce errors and make resident life safer, few authors have specifically analyzed the effect of the Accreditation Council for Graduate Medical Education (ACGME) duty-hour limits on neurosurgical resident education and the perceived quality of training. The authors have evaluated multiple objective surrogate markers of resident performance and quality of training to determine the impact of the 80-hour workweek.The United States Medical Licensing Examination (USMLE) Step 1 data on neurosurgical applicants entering ACGME-accredited programs between 1998 and 2007 (before and after the implementation of the work-hour rules) were obtained from the Society of Neurological Surgeons. The American Board of Neurological Surgery (ABNS) written examination scores for this group of residents were also acquired. Resident registration for and presentations at the American Association of Neurological Surgeons (AANS) annual meetings between 2002 and 2007 were examined as a measure of resident academic productivity. As a case example, the authors analyzed the distribution of resident training hours in the University of Virginia (UVA) neurosurgical training program before and after the institution of the 80-hour workweek. Finally, program directors and chief residents in ACGME-accredited programs were surveyed regarding the effects of the 80-hour workweek on patient care, resident training, surgical experience, patient safety, and patient access to quality care. Respondents were also queried about their perceptions of a 56-hour workweek.Despite stable mean USMLE Step 1 scores for matched applicants to neurosurgery programs between 2000 and 2008, ABNS written examination scores for residents taking the exam for self-assessment decreased from 310 in 2002 to 259 in 2006 (16% decrease, p < 0.05). The mean scores for applicants completing the written examination for credit also did not change significantly during this period. Although there was an increase in the number of resident registrations to the AANS meetings, the number of abstracts presented by residents decreased from 345 in 2002 to 318 in 2007 (7% decrease, p < 0.05). An analysis of the UVA experience suggested that the 80-hour workweek leads to a notable increase in on-call duty hours with a profound decrease in the number of hours spent in conference and the operating room. Survey responses were obtained from 110 program directors (78% response rate) and 122 chief residents (76% response rate). Most chief residents and program directors believed the 80-hour workweek compromised resident training (96%) and decreased resident surgical experience (98%). Respondents also believed that the 80-hour workweek threatened patient safety (96% of program directors and 78% of chief residents) and access to quality care (82% of program directors and 87% of chief residents). When asked about the effects of a 56-hour workweek, all program directors and 98% of the chief residents indicated that resident training and surgical education would be further compromised. Most respondents (95% of program directors and 84% of chief residents) also believed that additional work-hour restrictions would jeopardize patient care.Neurological surgery continues to attract top-quality resident applicants. Test scores and levels of participation in national conferences, however, indicate that the 80-hour workweek may adversely affect resident training. Subjectively, neurosurgical program directors and chief residents believe that the 80-hour workweek makes neurosurgical training and the care of patients more difficult. Based on experience with the 80-hour workweek, educators think that a 56-hour workweek would further compromise neurosurgical training and patient care in the US.
Graduate medical education
Cite
Citations (160)
Abstract Context Upon requests from osteopathic medical schools, the National Resident Matching Program (NRMP) Charting Outcomes were redesigned to include osteopathic medical school seniors beginning in 2018 and one joint graduate medical education (GME) accreditation system, the Accreditation Council for Graduate Medical Education (ACGME), formed in 2020. Objectives The goal of this study is to analyze the match outcomes and characteristics of osteopathic applicants applying to surgical specialties following the ACGME transition. Methods A retrospective analysis of osteopathic senior match outcomes in surgical specialties from the NRMP Main Residency Match data from 2020 to 2022 and the NRMP Charting Outcomes data from 2020 to 2022 was performed. Results For surgical specialties, results show matching increased as United States Medical Licensing Examination (USMLE) Step 2 CK (clinical knowledge) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2 CE (cognitive evaluation) scores increased along with the number of contiguous rankings (p<0.001). The greatest indication for matching looking at scores alone were those who scored greater than 230 on Step 2 CK compared to below (p<0.001) and above 650 on Level 2 CE (p<0.001). However, those who scored 240 (p=0.025) on Step 2 CK were just as likely to match as those who scored 250 (p=0.022) when compared to those who scored below those scores. Increasing research involvement had little to no significance with the likelihood of matching across most surgical subspecialties. Conclusions Our study demonstrates that there are unique thresholds for Step 2 CK scores, Level 2 CE scores, and the number of contiguous ranks for each surgical specialty that, when reached, are significantly associated with match success. Although certain board score delineations are linked with higher match success rates, the rates level off after this point for most surgical specialties and do not significantly increase further with higher scores. In addition, thresholds within contiguous ranks for increasing match likelihood exist and vary across surgical specialties. Overall, this study highlights that the quantitative metrics utilized to assess applicants lack the correlation reported historically, and the data presently available need to be more substantiated.
Graduate medical education
Osteopathy
Licensure
Cite
Citations (2)
Objective. To describe residency program compliance to a 60% pass rate and 80% eligibility standard outlined by the Accreditation Council for Graduate Medical Education (ACGME) on the American Board of Pediatrics Certifying Examination. The hypothesis is that larger programs will have higher pass rates. Methods. Pediatric residency programs were retrospectively evaluated from 2008 to 2010 regarding the ACGME standards. Simple linear regression was performed to see if program pass rates were dependent on program size. Results. A total of 162/163 (83.4%) programs had first-time examinee pass rates of 60%. A total of 179/193 (92.7%) programs satisfied the 80% eligibility standard. The Northeast performed lower than Midwest, Southern, and Western states ( P < .001). The West performed higher than the Northeast, Midwest, and Southern states ( p < 0.05). Simple linear regression showed that performance depends on program size ( P < .001). Conclusions. A majority of programs satisfy the minimum ACGME standards. Program performance is associated with program size and location. These findings may alter application patterns to pediatric residency programs.
Graduate medical education
Cite
Citations (16)
Educational measurement
Cite
Citations (99)
Introduction: The United States Medical Licensing Examination (USMLE) Step 1 score is one of the few standardized metrics used to objectively review applicants for residency. In February 2020 the USMLE program announced that the numerical Step 1 scoring would be changed to a binary (Pass/Fail) system. In this study we sought to characterize how this change in score reporting will impact the application review process for emergency medicine (EM) program directors (PD). Methods: In March 2020 we electronically distributed a validated anonymous survey to EM PDs at 236 US EM residency programs accredited by the Accreditation Council for Graduate Medical Education. Results: Of 236 EM PDs, 121 responded (51.3% response rate). Overall, 72.7% believed binary Step 1 scoring would make the process of objectively comparing applicants more difficult. A minority (19.8%) believed it was a good idea, and 33.1% felt it would improve medical student well-being. The majority (88.4%) reported that they will increase their emphasis on Step 2 Clinical Knowledge (CK) for resident selection, and 85% plan to require Step 2 CK scores at application submission time. Conclusion: Our study suggests most EM PDs disapprove of the new Step 1 scoring. As more objective data is peeled away from the residency application, EM PDs will be left to rely more heavily on the few remaining measures, including Step 2 CK and standardized letters of evaluation. Further changes are needed to promote equity and improve the overall quality of the application process for students and PDs.
Graduate medical education
Equity
Cite
Citations (8)