Adverse drug reaction or innocent bystander? A systematic comparison of statistical discovery methods for spontaneous reporting systems

PURPOSE: Spontaneous reporting systems (SRSs) are used to discover previously unknown relationships between drugs and adverse drug reactions (ADRs). A plethora of statistical methods have been proposed over the years to identify these drug-ADR pairs. The objective of this study is to compare a wide variety of methods in their ability to detect these signals, especially when their detection is complicated by the presence of innocent bystanders (drugs that are mistaken to be associated with the ADR, since they are prescribed together with the drug that is the ADR's actual cause). METHODS: Twelve methods, 24 measures in total, ranging from simple disproportionality measures (eg, the reporting odds ratio), hypothesis tests (eg, test of the Poisson mean), Bayesian shrinkage estimates (eg, the Bayesian confidence propagation neural network, BCPNN) to sparse regression (LASSO), are compared in their ability to detect drug-ADR pairs in a large number of simulated SRSs with varying numbers of innocent bystanders and effect sizes. The area under the precision-recall curve is used to assess the measures' performance. RESULTS: Hypothesis tests (especially the test of the Poisson mean) perform best when the associations are weak and there is little to no confounding by other drugs. When the level of confounding increases and/or the effect sizes become larger, Bayesian shrinkage methods should be preferred. The LASSO proves to be the most robust against the innocent bystander effect. CONCLUSIONS: There is no absolute "winner". Which method to use for a particular SRS depends on the effect sizes and the level of confounding present in the data.