Validating prediction models of kidney transplant outcome using single center data

: Prediction of kidney transplant outcome represents an important and clinically relevant problem. Although several prediction models have been proposed based on large, national collections of data, their utility at the local level (where local data distributions may differ from national data) remains unclear. We conducted a comparative analysis that modeled the outcome data of transplant recipients in the national US Renal Data System (USRDS) against a representative local transplant dataset at the University of Utah Health Sciences Center, a regional transplant center. The performance of an identical set of prediction models was evaluated on both national and local data to assess how well national models reflect local outcomes. Compared with the USRDS dataset, several key characteristics of the local dataset differed significantly (e.g., a much higher local graft survival rate; a much higher local percentage of white donors and recipients; and a much higher proportion of living donors). This was reflected in statistically significant differences in model performance. The area under the receiver operating characteristic curve values of the models predicting 1, 3, 5, 7, and 10-year graft survival on the USRDS data were 0.59, 0.63, 0.76, 0.91, and 0.97, respectively. In contrast, in the local dataset, these values were 0.54, 0.58, 0.58, 0.61, and 0.70, respectively. Prediction models trained on a national set of data from the USRDS performed better in the national dataset than in the local data. This might be due to the differences in the data characteristics between the two datasets, suggesting that the wholesale adoption of a prediction model developed on a large national dataset to guide local clinical practice should be done with caution.