Assessment of the Reliability of Data Collected for the Department of Veterans Affairs National Surgical Quality Improvement Program

Background The Office of the Medical Inspector of the Department of Veterans Affairs (VA) studied the reliability of data collected by the VA's National Surgical Quality Improvement Program (NSQIP). The study focused on case selection bias, accuracy of reports on patients who died, and interrater reliability measurements of patient risk variables and outcomes. Study Design Surgical data from a sample of 15 VA medical centers were analyzed. For case selection bias, reviewers applied NSQIP criteria to include or exclude 2,460 patients from the database, comparing their results with those of NSQIP staff. For accurate reporting of patients who died, reviewers compared Social Security numbers of 10,444 NSQIP records with those found in the VA Beneficiary Identification and Records Locator Subsystem, VA Patient Treatment Files, and Social Security Administration death files. For measurement of interrater reliability, reviewers reabstracted 59 variables in each of 550 patient medical records that also were recorded in the NSQIP database. Results On case selection bias, the reviewers agreed with NSQIP decisions on 2,418 (98%) of the 2,460 cases. Computer record matching identified 4 more deaths than the NSQIP total of 198, a difference of about 2%. For 52 of the categorical variables, agreement, uncorrected for chance, was 96%. For 48 of 52 categorical variables, kappas ranged from 0.61 to 1.0 (substantial to almost perfect agreement); none of the variables had kappas of less than 0.20 (slight to poor agreement). Conclusions This sample of medical centers shows adherence to criteria in selecting cases for the NSQIP database, for reporting deaths, and for collecting patient risk variables.