10
Citation
35
Reference
10
Related Paper
Citation Trend
Abstract:
Abstract Rationale, aims and objectives Meta‐analyses of diagnostic test accuracy are important elements in evidence‐based medicine. However, currently there is no overview of related quantitative findings that were obtained in a large number of real meta‐analyses. This study aimed at providing such empirical summary. Methods From the literature 50 meta‐analyses were randomly selected that had reported their 2 × 2 count data of sensitivity and specificity. Descriptive statistics, assessment of between‐study heterogeneity and bivariate random‐effects meta‐analysis of sensitivity and specificity were performed with a novel B ayesian program code. The bivariate model parameters were also converted to the parameters of the closely related hierarchical summary receiver operating characteristic ( HSROC ) model. Results Among the 50 meta‐analyses, the studies per meta‐analysis ranged from 5 to 45 and the disease prevalence from 2.3 to 71%. Significant between‐study heterogeneity was found in 43 of 50 meta‐analyses, favouring a random‐effects model over a fixed‐effects model. Empirical distributions of sensitivity and specificity, positive and negative likelihood ratios, and other model results are presented in the full text numerically and graphically. Conclusions Studies of diagnostic test accuracy can be well meta‐analysed within a B ayesian framework, and the presented quantitative findings provide an orientation when interpreting the results of the standard bivariate/ HSROC model.Meta-analyses of diagnostic accuracy studies are a fundamental component of evidence-based medicine, and they are extensively used in medical imaging and the clinical laboratory. Techniques specifically developed to combine independent studies of diagnostic accuracy and provide pooled estimates for sensitivity (Se), specificity (Sp), positive (pLR) and negative (nLR) likelihood ratios are relatively new. In 2001, Rutter and Gatsonis proposed the hierarchical summary receiver operating characteristic (HSROC) model,1 and in 2004 Macaskill described an empirical Bayes approach.2 Soon after, in 2005, Reitsma et al. proposed the bivariate random effects model,3 which has been widely adopted and is the most commonly used method for diagnostic meta-analysis.4 However, as pointed out by Diaz,5 the statistical performance of the bivariate model has not been scrutinized. Diaz found that the performance of the bivariate model deteriorates when between-study heterogeneity increases and the number of studies decrease.5 Our simulation studies found similar results—with moderate levels of heterogeneity (tau2 = 1), the coverage probabilities of Se, Sp, and the diagnostic odds ratio (DOR) with the bivariate model dropped below the nominal level.6 Diagnostic accuracy studies usually favor sensitivity over specificity, or vice versa leading to diagnostic 2 × 2 tables with one or more of the cells with low frequency or zero counts. Thus, extreme DORs are more commonly observed in diagnostic than in intervention meta-analyses, which leads to high levels of heterogeneity (despite the wide confidence intervals of the studies).7 The analyses were conducted in Stata MP version 14.1 using the metandi module12 for the bivariate models and the diagma module13 for the SCS method. The Bayes-HSROC model was implemented in the R programming language using the rjags14 and runjags15 packages. The Bayes-HSROC model applies Bayesian inference, where the posterior distribution of the parameters of interest depends on the likelihood function and the prior information provided. The likelihood function was computed as a statistical model for the observed data (Supplementary Material S1). Noninformative priors were used, meaning that no external information was provided to the model. Parameter estimates were based on analytical summaries of 500,000 iterations of two chains after a burn-in phase of 10,000 iterations. Time series plots were used to assessed convergence.16 The two chains converged to the same solution and autocorrelation plots dropped-off fast (Supplementary Material S2). The results of the four models are presented in Table 2, and point estimates and confidence intervals (credible intervals for the Bayes-HSROC model) were more conservative with the SCS methods and the Bayes-HSROC model than with both bivariate models. The Se was 98.4 (95% CI 90.2–99.8) with the bivariate model, while it was 95.6 (95% CI 62.6–99.6) and 92.7 (95% CI 67.4–99.8) with the SCS method and Bayesian HSROC respectively, with similar results for the Sp. The five studies included in the case study were simulated fixing the sample size to original study and fixing the true value of Se and Sp = 0.96 (based on the pooled estimates in Table 2). The number of diseased (dis) and nondiseased (ndis) individuals were drawn from a binomial distribution using the sample size and the actual prevalence of seropositivity and seronegativity, respectively in each study. The four cell counts (tp, fp, fn, tn) were then derived from dis and ndis, and the Se and Sp. Next, the four counts were divided by a scale parameter (minimum value = 1) that was derived from a transformation of a hypothetically imputed bias variance to introduce systematic error.17 Both random and systematic error were introduced by regenerating a simulated Se and Sp from a beta distribution with parameters tp/f and fn/f; and tn/f and fp/f, respectively. Next, the studies were generated and meta-analyzed, and 1000 meta-analyses were simulated in each of 10 runs, with run 1 representing random error alone (scale parameter = 1) and runs 2–10 having increasing level of between-study heterogeneity. The Stata codes for the data generation are provided in the Supplementary Material S3. For each level of heterogeneity, summary DOR, Se, and Sp estimated by the extension of the bivariate model (proposed by Chu and Cole) and SCS method were compared based on mean absolute estimation error squared (bias squared), mean squared error (MSE), width of the confidence interval, and coverage probability.18 The distribution of Se, Sp, and tau2 generated for each of the 10 runs are reported in the Supplementary Material S4. The bivariate model did not converge in 19% of the simulated meta-analyses and these were excluded from the performance analyses for both the bivariate model and SCS method. The simulation study revealed that the SCS method's DOR, Se, and Sp estimates were less biased (Figure 1A) and had smaller MSE than the bivariate model estimates (Figure 1B). As heterogeneity increased, the width of the 95% confidence interval became wider with the bivariate model (Figure 1C), yet it had lower coverage probability of the confidence interval compared to the SCS method (Figure 1D). It was not possible to compare the performance of the models when moderate or extensive heterogeneity was introduced as the bivariate model did not converge in > 50% of the meta-analyses. In our case study with small number of studies and large heterogeneity, discrepancies were observed in the confidence/credible intervals - very narrow confidence intervals with the bivariate models, while the confidence/credible intervals were wide with the SCS method and the Bayes-HSROC. The simulation study revealed that when heterogeneity was introduced, there was a considerable decline in the performance of the bivariate model. Therefore, it is very likely that the results of the case study and other studies using the bivariate model would generate spuriously overconfident results due to overdispersion of the data relative to the model. Between-study heterogeneity is the norm in meta-analyses of diagnostic accuracy studies. In a methodological review, Dinnes et al.19 found that there was statistical heterogeneity in 79% of diagnostic meta-analyses; thus pooling methods have to be able to properly maintain performance when heterogeneity is present. This study therefore suggests that newer SCS method can resolve the issue of overdispersion with the bivariate model and needs to be prioritized in research. Alternatively, a Bayesian approach can be used, especially when the reference method is imperfect. In conclusion, the bivariate model suffers from the same issue of overdispersion as the random effects model in standard meta-analysis20 and the SCS method seems to be a viable alternative. The latter also avoids the issue of nonconvergence and is not unduly affected by varying implicit thresholds given that it starts with synthesis of the DOR. Further evaluation is therefore recommended to independently verify these findings, so that the necessary recommendations can be made for the research community Open Access Funding provided by The University of Queensland. LFK was supported by Australian National Health and Medical Research Council Early Career Fellowships (APP1158469). The data that supports the findings of this study are available in the Supplementary Material of this article. Supplementary Material S1. R/JAGS code to run Bayes-HSROC model Supplementary Material S2. Time series plots for (A) pooled sensitivity, (B) pooled specificity, (C) reference method sensitivity, (D) reference method specificity Supplementary Material S3. Stata code for data simulation Supplementary Material S4. Summary output of the simulation study Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
Diagnostic odds ratio
Cite
Citations (4)
In meta-analyses involving a few trials, appropriate measures should be employed to assess between-study heterogeneity. When the number of studies is less than five and heterogeneity is evident, the Hartung and Knapp (HK) correction should be used. The aim of this study was to compare the reported estimates of published orthodontic meta-analyses with the pooled effect size estimates and prediction intervals (PI) calculated using eight heterogeneity estimators and corrected using the HK correction.
Pooled variance
Study heterogeneity
Interquartile range
Forest plot
Cite
Citations (3)
Mixed model
Cite
Citations (116)
One of the reasons for the popularity of meta-analysis is the notion that these analyses will possess more power to detect effects than individual studies. This is inevitably the case under a fixed-effect model. However, the inclusion of the between-study variance in the random-effects model, and the need to estimate this parameter, can have unfortunate implications for this power. We develop methods for assessing the power of random-effects meta-analyses, and the average power of the individual studies that contribute to meta-analyses, so that these powers can be compared. In addition to deriving new analytical results and methods, we apply our methods to 1991 meta-analyses taken from the Cochrane Database of Systematic Reviews to retrospectively calculate their powers. We find that, in practice, 5 or more studies are needed to reasonably consistently achieve powers from random-effects meta-analyses that are greater than the studies that contribute to them. Not only is statistical inference under the random-effects model challenging when there are very few studies but also less worthwhile in such cases. The assumption that meta-analysis will result in an increase in power is challenged by our findings.
Statistical power
Popularity
Power analysis
Statistical Analysis
Cite
Citations (482)
We read with interest the systematic review and meta-analysis by Li et al.1 published in the Journal of Paediatrics and Child Health. The study describes a meta-analysis which tries to show whether sleep duration is associated with higher risk of obesity. It included 12 prospective cohort studies involving about 44 200 participants. The study concluded that short sleep duration is associated with a 45% increased risk of subsequent obesity in children.1 Taking into account certain statistical issues in this meta-analysis, the pooled relative risk (RR) of the association between obesity and short sleep duration had some level of underestimation. The underestimation stemmed from a common problem in meta-analysis that the random effects model was employed in the presence of significant publication bias. The authors in the field of meta-analysis have a common conception that random effects model meta-analysis yield more conservative estimates than fixed effect model, especially when there is observable heterogeneity among studies. But, we can discuss that in the random effect model attributed weights are very similar across studies with different sample size and different precision, therefore the pooled estimation derived from a random effect model can be highly biased especially in the presence of publication bias.2, 3 In opposite direction, the fixed effect model allocate more weight to larger studies.2 In such a scenario, pooled RR from a fixed effect model is more conservative in the presence of publication bias. Therefore, we reanalyzed the paper information to estimate how selection of a random effect model in the presence of publication bias can affect Li et al.’s reported results. Contour-enhanced funnel plot was used for assessing the presence of publication bias. As Figure 1 shows, there is a strong asymmetry in the plot; studies with larger RR (and less precision) clustered at the bottom right of the pooled log RR in the plot. This result in together with significant egger's publication bias test (b = −2.41, P = 0.03) indicate observable publication bias in this study. Considering the presence of publication bias, we used a fixed effect model and the pool estimate was RR 2.14 (95% CI 2.10–2.18) (Fig. 2). Indeed, fixed effect model estimation indicates that the risk of obesity can be 200% in children with short sleep duration. But, we should bear in mind that the fixed effect model presented gives almost all weight to one study, and the pooled RR is therefore rather a report of the result of one large study (93.2% weight allocated to one cohort study in Fig. 2) than a meta-analysis of all available evidence. On the other view, we can discuss that there is little evidence for having a conclusive estimation for the association between sleep duration and obesity in children. In conclusion, our result based on fixed effect model is similar with Li et al.1 and both indicate a medium positive effect, but the two analyses disagree on the size of the pooled RR. Conclusive interpretation of the magnitude of the effect needs further large scale prospective cohort studies.
Cite
Citations (3)
Whether environmental exposure to Manganese (Mn) in adults is associated with poorer results in cognitive and motor function is unclear. We aimed to determine these associations through a meta-analysis of published studies.A systematic review was conducted to identify epidemiological studies on a population ≥18 years old exposed to environmental airborne Mn, and in which results on specific tests to evaluate cognitive or motor functions were reported. We consulted Medline through PubMed, Web of Science and SCOPUS databases. We also performed a manual search within the list of bibliographic references of the retrieved studies and systematic reviews. To weight Mn effects, a random effects versus fixed effect model was chosen after studying the heterogeneity of each outcome.Eighteen studies met the inclusion criteria. Among them, eleven studies reported data susceptible for meta-analysis through a pooled correlation or a standardized means difference (SMD) approach between exposed and non-exposed groups. Regarding cognitive function, the results of the studies showed heterogeneity among them (I2 = 76.49%, p < 0.001). The overall effect was a statistically significant negative correlation in the random effects model (pooled r = -0.165; 95%CI: -0.214 to -0.116; p < 0.001). For SMD, the results showed a lower heterogeneity with a negative SMD that did not reach statistical significance under the fixed effects model (SMD = -0.052; 95%CI -0.108 to 0.004; p = 0.068). Regarding motor function, heterogeneity (I2 = 75%) was also observed in the correlation approach with a pooled r (random effect model) = -0.150; 95%CI: -0.219 to -0.079; p < 0.001. Moderate heterogeneity was observed according to the SMD approach (I2 = 52.28%), with a pooled SMD = -0.136; 95%CI: -0.188 to-0.084; p < 0.001, indicating worse motor function in those exposed.Correlation approach results support a negative effect on cognitive and motor functions (the higher the Mn levels, the poorer the scores). Regarding the SMD approach, results also support a worse cognitive and motor functions in those exposed, although only for motor function statistical significance was obtained.
Web of science
Study heterogeneity
Pooled variance
Cite
Citations (10)
BackgroundMultivariate meta-analysis is used when multiple correlated outcomes are reported in a systematic review. This study explored the application of multivariate meta-analysis in such a context. The objectives of the present study were to compare the summary findings and decisions between univariate and bivariate meta-analyses, as well as to assess how much sensitive the results are towards the strength of the correlation between the outcome variables.MethodsA systematic review that reported two correlated outcomes, Intact parathyroid hormone levels and serum phosphate was chosen for demonstrating the applications of bivariate meta-analysis. Both univariate and bivariate meta-analyses with fixed effect and random effect models were carried out and the results were compared. A sensitivity analysis was performed for a wide spectrum of correlations from −1 to +1 to assess the impact of correlation on pooled effect estimates and its precision.ResultsPooled effect estimates generated through bivariate meta-analysis were found to be varying when compared to those obtained through univariate meta-analysis. The confidence interval of the pooled effect estimates obtained through bivariate meta-analysis was wider than in univariate meta-analysis. Further, the value of the pooled effect estimates along with its confidence intervals also differed for varied levels of correlations.ConclusionsThis study observed that when we have multiple correlated outcome variables to answer a single question bivariate meta-analysis could be a better approach. The magnitude of the correlation between the outcome variables also plays a vital role in meta-analysis.
Univariate
Pooling
Univariate analysis
Cite
Citations (1)
Abstract Rationale, aims and objectives Meta‐analyses of diagnostic test accuracy are important elements in evidence‐based medicine. However, currently there is no overview of related quantitative findings that were obtained in a large number of real meta‐analyses. This study aimed at providing such empirical summary. Methods From the literature 50 meta‐analyses were randomly selected that had reported their 2 × 2 count data of sensitivity and specificity. Descriptive statistics, assessment of between‐study heterogeneity and bivariate random‐effects meta‐analysis of sensitivity and specificity were performed with a novel B ayesian program code. The bivariate model parameters were also converted to the parameters of the closely related hierarchical summary receiver operating characteristic ( HSROC ) model. Results Among the 50 meta‐analyses, the studies per meta‐analysis ranged from 5 to 45 and the disease prevalence from 2.3 to 71%. Significant between‐study heterogeneity was found in 43 of 50 meta‐analyses, favouring a random‐effects model over a fixed‐effects model. Empirical distributions of sensitivity and specificity, positive and negative likelihood ratios, and other model results are presented in the full text numerically and graphically. Conclusions Studies of diagnostic test accuracy can be well meta‐analysed within a B ayesian framework, and the presented quantitative findings provide an orientation when interpreting the results of the standard bivariate/ HSROC model.
Cite
Citations (10)
The primary purpose of this paper is to comprehensively assess households' burden due to health payments. Starting from the fairness approach developed by the World Health Organization, we analyse the burden of healthcare payments on Italian households by modeling catastrophic payments and impoverishment due to healthcare expenditures. For this purpose, we propose to extend the analysis of fairness in financing contribution through a generalized linear mixed models by introducing a bivariate correlated random effects model, where association between the outcomes is modeled through individual- and outcome-specific latent effects which are assumed to be correlated. We discuss model parameter estimation in a finite mixture context. By using such model specification, the fairness of the Italian national health service is investigated.
Cite
Citations (10)
Background/Aim. Whether environmental exposure to Mn in adults is associated with poorer results in cognitive and motor function is unclear. We aimed to determine these associations through a meta-analysis of published studies. Methods: A systematic review was conducted to identify environmental Mnepidemiologic studies in >=18 years old, and in which results on a specific test to evaluate cognitive or motor function were reported. Medline through PubMed, ISI Web of Knowledge and SCOPUS databases were consulted. A manual search in the references of retrieved studies and systematic reviews found that addressed the topic was also conducted. Data were pooled in meta-analysis using the method of random effects or fixed effects, as convenient, after examination of statistical heterogeneity. Results.Seventeen studies fulfill inclusion criteria. Among them, 13 studies reported data susceptible of meta-analysis through a pooled correlation or a Standardized Means Difference (SMD) approach between exposed and non-exposed. Regarding cognitive function, the results of the studies showed heterogeneity among them (I2=76.49%, p<0.001). The overall effect was a statistically significant negative correlation in the random effects model (pooled r=-0.165; 95%CI: -0.214 to -0.116; p<0.001). In terms of SMD, results showed also moderate heterogeneity but did not reach statistical significance under the random effects model (SMD=-0.049; 95%CI: -0.124 to 0.026; p=0.203). Regarding motor function, heterogeneity (I2=75%)was also observed in the correlation approach with a pooled r (random effect model)=-0.150; 95%CI: -0.219 to -0.079. Moderate heterogenety was observed according to SMD approach (I2=51.81%), with a pooled SMD=-0.136; 95%CI: -0.188 to-0.084; p<0.001, indicating worse motor function in exposed. Conclusions: Correlation approach results support a negative effect on cognitive and motor function (the higher the Mn levels, the poorer scores). Regarding SMD approach, results also support a worse cognitive and motor function in exposed, although only for motor function statistical significance was obtained.
Study heterogeneity
Pooled variance
Cite
Citations (0)