Not all risk scores are created equal: A comparison of risk scores for abdominal aortic aneurysm repair in administrative data and quality improvement registries.

Abstract Objective Accurate and contemporary prognostic risk prediction is essential to inform clinical decision-making surrounding abdominal aortic aneurysm (AAA) care. Therefore, we validated and compared three different in-hospital mortality risk scores in one administrative and two quality improvement registries. Methods We included patients who had undergone elective AAA repair from 2012 to 2015 in the National Inpatient Sample (NIS), Vascular Quality Initiative (VQI; excluding the New England region), and the National Surgical Quality Improvement Program (NSQIP) datasets to validate three risk scores: Medicare, the Vascular Study Group of New England (VSGNE), and Glasgow Aneurysm Score (GAS). The receiver operating characteristic area under the curve (AUC) of all risk scores was calculated, and their discrimination was compared within a dataset using the Delong test and between datasets using a Z test. We constructed graphic calibration curves for the Medicare and VSGNE risk scores and compared the calibration using an integrated calibration index, which indicates the weighted average of the absolute difference between the calibration curve and the diagonal line of perfect calibration. Results We identified a total of 25,461 NIS, 18,588 VQI, and 8051 NSQIP patients who had undergone elective open or endovascular AAA repair. Overall, the Medicare risk score was more likely to overestimate mortality in the quality improvement registries and the VSGNE risk score underestimated mortality in all the databases. After endovascular AAA repair, the Medicare risk score had a higher AUC in the NIS than in the GAS (P  Conclusions Overall, the VSGNE risk score performed best in the quality improvement registries but underestimated mortality. However, the Medicare risk score demonstrated better calibration in the administrative dataset after open repair. Although the VSGNE risk score appeared to perform better in the quality improvement registries, its overly optimistic mortality estimates and its reliance on detailed anatomic and clinical variables reduces its broader applicability to other databases.