A validated search assessment tool: assessing practice-based learning and improvement in a residency program

Objective: The objective of this study was to validatean assessment instrument for MEDLINE searchstrategies at an academic medical center.Method: Two approaches were used to investigate ifthe search assessment tool could capture performancedifferences in search strategy construction. First, datafrom an evaluation of MEDLINE searches from apediatric resident’s longitudinal assessment wereinvestigated. Second, a cross-section of searchstrategies from residents in one incoming class wascompared with strategies of residents graduating ayear later. MEDLINE search strategies formulated byfaculty who had been identified as having searchexpertise were used as a gold standard comparison.Participants were presented with a clinical scenarioand asked to identify the search question and conducta MEDLINE search. Two librarians rated the blindedsearch strategies.Results: Search strategy scores were significantlyhigher for residents who received training than thecomparison group with no training. There was nosignificant difference in search strategy scoresbetween senior residents who received training andfaculty experts.Conclusion: The results provide evidence for thevalidity of the instrument to evaluate MEDLINEsearch strategies. This assessment tool can measureimprovements in information-seeking skills andprovide data to fulfill Accreditation Council forGraduate Medical Education competencies.INTRODUCTIONThe acquisition of strong information-searching skillsis one of the many building blocks upon whichmedical knowledge and clinical skills are built. TheAccreditation Council for Graduate Medical Educa-tion (ACGME) has listed practice-based learning andimprovement (PBLI) as a core competency, definingPBLI as the ability to (1) investigate and evaluate thecare of patients, (2) appraise and assimilate scientificevidence, and (3) continuously improve patient carebased on constant self-evaluation and lifelong learn-ing [1]. Of these, the ability to appraise and assimilatescientific evidence is infrequently taught in manypostgraduate medical education programs, and cur-rently no one standardized method is used to assessthis competency [2].Being able to search the medical literature effec-tively is an essential skill in the practice of evidence-based medicine (EBM). Moreover, developing andmaintaining strong literature searching skills (use ofappropriate search keywords, use of appropriatelimits, development of evidence-based search strate-gies, and use of correct search syntax) help physiciansidentify appropriate literature that affects medicaldecision making [3]. Due to their training in the areasof MEDLINE search strategies and search retrieval,medical librarians are recognized as specialists inbiomedical literature searching and search assessment[4].Historically, a variety of methods and tools havebeen used to assess residents’ search skills [5].Instruments have typically been based on the devel-opment of a gold standard search strategy, asdetermined by an expert searcher, against which allresidents’ searches for a specific clinical question arecompared. The comparison includes identified ele-