Between-trial heterogeneity in meta-analyses may be partially explained by reported design characteristics

Abstract Objective We investigated the associations between risk of bias judgments from Cochrane reviews for sequence generation, allocation concealment and blinding, and between-trial heterogeneity. Study Design and Setting Bayesian hierarchical models were fitted to binary data from 117 meta-analyses, to estimate the ratio λ by which heterogeneity changes for trials at high/unclear risk of bias compared with trials at low risk of bias. We estimated the proportion of between-trial heterogeneity in each meta-analysis that could be explained by the bias associated with specific design characteristics. Results Univariable analyses showed that heterogeneity variances were, on average, increased among trials at high/unclear risk of bias for sequence generation ( λ ˆ 1.14, 95% interval: 0.57–2.30) and blinding ( λ ˆ 1.74, 95% interval: 0.85–3.47). Trials at high/unclear risk of bias for allocation concealment were on average less heterogeneous ( λ ˆ 0.75, 95% interval: 0.35–1.61). Multivariable analyses showed that a median of 37% (95% interval: 0–71%) heterogeneity variance could be explained by trials at high/unclear risk of bias for sequence generation, allocation concealment, and/or blinding. All 95% intervals for changes in heterogeneity were wide and included the null of no difference. Conclusion Our interpretation of the results is limited by imprecise estimates. There is some indication that between-trial heterogeneity could be partially explained by reported design characteristics, and hence adjustment for bias could potentially improve accuracy of meta-analysis results.