On the criteria used for assessing the risk of bias in randomized trials included in systematic reviews and meta-analyses addressing adverse effects

Systematic reviews and meta-analyses are a valuable tool to answer research questions, generate recommendations and influence health policy in an evidence-based manner [1]. However, flaws in the design, conduct, analysis, and reporting of the studies that are included in a systematic review may distort estimates of the true intervention effect in terms of magnitude and/or direction (i.e. introduce bias), mislead the analysis and conclusions of the systematic review, and adversely influence clinical practice and health policy decisions [2]. Based on a combination of empirical and theoretical considerations, the Cochrane Statistical Methods Group and the Cochrane Bias Methods Group proposed a set of criteria for assessing the risk of bias in RCTs, which address the following key domains: (1) sequence generation, (2) allocation concealment, (3) blinding of participants, personnel, and outcome assessors, (4) incomplete outcome data, (5) selective outcome reporting, and (6) other sources of bias (e.g., extreme baseline imbalance in prognostic factors, inappropriate influence of the funding source, etc) [3]. The Cochrane Collaboration’s domain-based approach has proven to be very useful; it has been widely adopted, and used in hundreds of systematic reviews and metaanalyses. However, the Cochrane Adverse Effects Methods Group proposed few additional criteria for assessing the risk of bias in RCTs selected for inclusion in systematic reviews addressing adverse effects (AEs) [4] including methods for monitoring and detecting adverse effects [4] (p. 8). The particular item for judging risk of bias was: Were the methods used for monitoring adverse effects reported? Use of prospective or routine monitoring; spontaneous reporting; patient checklist, questionnaire or diary; systematic survey of patients? [4] (p. 9). As a result, this criterion is increasingly used in the literature, with researchers considering studies providing spontaneous reporting of AEs (passive surveillance) as of higher risk of bias than studies actively monitoring for drug AEs [5–8]. We argue that this proposed risk-of-bias criterion (based on the approach used for monitoring AEs) is methodologically incorrect, and could lead reviewers to assess potentially fair (unbiased) clinical trials as having a high risk of bias. It is true that the methods used for monitoring AEs have a major influence on the detected event rates in a trial, with passive surveillance of harms leading to fewer recorded AEs than active surveillance [9, 10]. However, the relative effect measure in a trial (e.g. the ratio of the observed event rates in the comparison groups) will not change if these event rates are underestimated, to the same degree, in each comparison group. Therefore less intensive monitoring methods, although they may lead to lower observed rates of AEs, do not constitute a source of bias, but rather a cause of imprecision. A potential source of bias must be able to produce results that depart systematically & Stefanos Bonovas sbonovas@gmail.com