Which patients benefit most from completing health risk assessments: comparing methods to identify heterogeneity of treatment effects

Methods for identifying heterogeneity of treatment effects in randomized trials have seen recent advances, yet applying these methods to health services intervention trials has not been well investigated. Our objective was to compare two approaches—predictive risk modeling and model-based recursive partitioning—for identifying subgroups of trial participants with potentially differential response to an intervention involving health risk assessment completion alone (n = 192) versus health risk assessment completion plus telephone-delivered health coaching (n = 173). Notably, these approaches have been developed by investigators from distinct disciplines and reported in separate literatures and have generally not been compared in prior work. Furthermore, these methods approach subgroup identification differently and answer related but slightly different questions. The primary outcome for both approaches was prevention health program enrollment by six months. The predictive risk model was developed in two steps, where, first, a single risk score was derived from a logistic regression model with 12 a priori chosen covariates by the scientific investigator team (c-statistic = 0.63). Then, the treatment effect was calculated within quartiles of risk via interaction in a logistic regression model (c-statistic = 0.69; c-for-benefit = 0.43). The greatest treatment effect was in the second quartile, in which 54% (22 of 41) of intervention patients and 10% (5 of 50) of control patients reported prevention program enrollment. In contrast, with the data-driven approach of model-based recursive partitioning, all 28 baseline covariates were considered, with the algorithm selecting covariates and optimal split points. Final model results had a c-statistic of 0.69 and a c-for-benefit of 0.55 (optimism-corrected c-statistic = 0.62 and c-for-benefit = 0.53) and identified 4 subgroups, with the greatest treatment effect among patients with lower mean numeracy, education less than a bachelor’s degree, and diabetes, in which 54% (15 of 28) of intervention patients reported prevention program enrollment versus 7% (3 of 41) of control patients. While there is increasing interest in discovering heterogeneity of treatment effects, our analyses highlight the important differences between these approaches, both from questions answered, model development, and results obtained. Specifying goals of treatment heterogeneity analyses, choosing the appropriate method to best address the goals, and external validation of results are important steps when applying these methods. Clinicaltrials.gov identifier: NCT01828567