Use of Uniform Outcomes Methodologies to Measure Clinical Impact of Large-Scale Continuing Medical Education Initiatives

Background: The purpose of this discussion is to describe the benefits of a uniform approach to measuring clinical outcomes and to report on the outcomes results for a sample of live Continuing Medical Education (CME) sessions. Demonstrating measurable and meaningful outcomes in physician knowledge and practice is an extremely important opportunity for providers of CME. Since 2002, Pri-Med has been using a wide variety of uniform tools to measure outcomes, including session and speaker evaluation forms, question-and-answer cards, interactive audience response data, and control/participant long-term knowledge retention instruments. The data collected is then analyzed univariately and multivariately (eg, regression). However, the power of this outcomes measurement is the ability to benchmark and compare results against a normative database of previous activities. Outcomes are evaluated from an absolute and comparative basis by the Pri-Med Outcomes Advisory Board to provide historical trends and future directions for educational programs that address knowledge gaps among physicians. Method: In 2005, the Pri-Med Outcomes Study evaluated the clinical outcomes for a sample of 893 live CME sessions, across 35,452 clinicians who completed the pre-education assessment, yielding an average response rate of 11%, and 19,511 clinicians who completed the post-education assessment with an average response rate of 8%. The sessions evaluated covered 48 disease states and were conducted in 49 cities in the United States. The objectives of the outcomes study were to gauge the effectiveness of each session’s ability to enhance attendees’ knowledge and understanding of the clinical area presented by measuring a change in 3 areas, referred to here as metrics : (1) adherence to guideline; (2) confidence in treatment; and (3) knowledge gained. At the end of each session, evaluation data and qualitative feedback were collected via an on-site evaluation form distributed to participants. In addition, pre- and 6-week post-education measurement tools were administered via e-mail to pre-registered and on-site and verified session attendees. The pre-education assessment can be defined as the “control,” “baseline,” or those clinicians that had registered for the session but had not yet been exposed to the education. The post-education assessment can be defined as the “participant,” “test” group, or those who attended the session and had been exposed to the education. Pre- and post-education results were analyzed for each individual session using Pri-Med’s proprietary Educational Impact Index—a cumulative measure of the 3 leading outcomes metrics at a 95% significance level ( P = .05). Results: Up to 6 weeks after each program, 99% of respondents reported having used the education obtained specifically to change or refine their approach to patient care. There was a significant increase in the Educational Impact Index (EII) calculated for all disease states measured. Specifically, the overall EII increased by 10%, from a score of 1.55 (pre-education) to 1.70 (post-education). Compared to the pre-education baseline results, post-education surveys scores indicated a 12% increase in adherence to guidelines, an 11% increase in confidence in treatment, and a 17% increase in knowledge ( P = .05). Conclusions: Implementing a system-wide approach for measuring outcomes provides a foundation from which to assess persistent learning across distinct physician profiles and topic areas. This leads to more effective, targeted education, by identifying clinician knowledge gaps and enabling a roadmap for future educational programs. The Pri-Med Outcomes Study demonstrates the benefits of using uniform measurement tools across programs to document the clinical impact of each CME initiative. These uniform measurement tools can be applied for ongoing outcomes assessments as well as any custom outcomes studies conducted on behalf of CME supporters. In 2006, Pri-Med has applied the learning and experience from 4+ years of measuring live outcomes activities to develop measurement tools for both online and print as well as for multi-faceted, multi-format CME curricula.