How robust are health plan quality indicators to data loss? A Monte Carlo simulation study of pediatric asthma treatment.

Quality Indicators (QIs) have become benchmarks for evaluating health plan performance. Typically, QIs are based on evidence-driven measures of quality of care for plan populations as a whole (e.g., percentage of members receiving an annual flu shot) or subpopulations with particular diseases (e.g., percentage of diabetics receiving annual eye exams). The Health Plan Employer Data and Information Set (HEDIS) measures developed by the National Committee for Quality Assurance (NCQA) stand at the forefront of health plan quality measures. The HEDIS measures are designed to permit a systematic and standardized approach to plan performance measurement. Depending upon the type of measure, health plans can choose to use administrative data (primarily, encounter data), medical record reviews, or client surveys to calculate these measures. Most plans find encounter data to be most efficient to use. However, several studies have reported that health plans face persistent problems in obtaining complete encounter data (Gold et al. 1995; Aizer, Felt, and Nelson 1996; National State Auditors Association 2002). In a survey of 108 managed care plans from 20 metropolitan areas nationwide, Gold et al. (1995) found that less than a quarter of plans received 90 percent or more of encounter data from their contracted physicians. A recent audit of four Medicaid managed care programs by the National State Auditors Association (2000) found that about one-third of the encounter data were lost. These circumstances raise an obvious question of how valid HEDIS measures are if the underlying data sources from which they are computed are incomplete. One important factor affecting the robustness of a QI measure to data loss is the manner in which the measure is constructed. In a study by Dresser et al. (1997), HEDIS rates for cervical cancer screening from administrative data were very close to those obtained from chart reviews, whereas the rates for pediatric immunization and prenatal care differed greatly. In the case of cervical cancer screening, the authors speculated that because both the clinician who performs the Pap test and the clinician who reads it may have recorded the event, the likelihood that true events are accurately measured is high even in the presence of data loss; that is, redundancy improves validity. On the other hand, the robustness of a QI measure to data loss is reduced if it is based on multiple events. In the Dresser study, for example, the QI measure for cancer screening required just one Pap test over three years but the QI measure for pediatric immunizations required nine visits over a two-year period. Many HEDIS measures are computed as proportions of patients receiving appropriate care. Examples include beta blocker treatment after heart attack, cholesterol management after acute cardiovascular event, comprehensive diabetes care (annual eye exam, hemoglobin A1c testing, lipid screening, nephropathy monitoring), follow-up after hospitalization for mental illness, use of appropriate medications in those with persistent asthma, and prenatal and postpartum care (http://www.ncqa.org). Each of these measures requires identification of a particular group of individuals, constituting the denominator of the proportion, based upon service encounter information (e.g., a prescription for insulin for identifying diabetics). The rate of a specific QI is then determined based upon another service encounter or encounters (e.g., eye exam), which constitutes the numerator of the proportion. Proportion-based QI measures are inherently more robust to data loss than count-based measures because any loss is likely to reduce the value of both the numerator and the denominator. In the special case where data loss is proportional in the numerator and denominator, the measure itself will be unaffected by the loss. There are, however, both empirical and mathematical reasons to suspect that proportion-based measures will degrade in the presence of data loss, particularly if the loss is severe. For example, if the numerator is a relatively rare event compared to the denominator, a given absolute loss of data will have a proportionally greater impact on the numerator, driving the fraction downward. On the other hand, as noted in the Dresser study, redundancy in the numerator makes the measure less sensitive to data loss. This article examines the robustness of a particular proportion-based quality indicator to various threats of data loss. We also assess the implications of the findings for other proportion-based HEDIS measures. The motivation for the analysis was a study of changes in quality of asthma treatment for a population of Medicaid children transitioning from fee-for-service (FFS) to managed care. The quality indicator was based on FFS claims in the before period and encounter data in the follow-up period. We had high confidence in the completeness of claims data, but much less confidence with the encounter data. Our challenge was to determine the validity of our quality indicator in the face of presumed, but unknown levels of data loss from the managed care plans. The remainder of the paper is organized as follows. The next section describes traditional approaches to dealing with missing data problems and explains why these did not address our particular concerns. Sections describing the study setting, methods, and results follow. We close with a discussion of the applicability of our methods for assessing the robustness of other HEDIS quality indicators to data loss.