Abstract Introduction Overlap of primary studies among systematic reviews (SRs) included in an overview is a major challenge, as it may bias results or artificially increase the precision of the synthesis. Matrices of evidence and corrected covered area (CCA) calculation are recommended methods to manage overlap, but there is little guidance on how to construct these matrices. This research aims to explore variations in the estimation of overlap using CCA matrices under different assumptions. Methods We will include overviews published in 2023. We will describe the methods used by authors to deal with overlap, and we will calculate a summary CCA (a CCA for the whole matrix of evidence) and a pairwise CCA (a CCA for each possible pair of included SRs), comparing the results under different assumptions that may modify the evidence matrix and thus the CCA. These assumptions include: publication-thread adjustments (i.e. the consideration of each set of references regarding a single primary study as a unique row in a matrix of evidence), scope adjustments (i.e. the consideration only of the SRs and primary studies providing useful data for a given outcome within an overview) and chronological structural missingness adjustments (i.e. the exclusion of primary studies published after a given SR for purposes of CCA calculation). We will assess overlap at an overview and outcome level. Discussion We propose clear definitions for the key assumptions for creating matrices of evidence. We expect to provide a guide for overview authors to better interpret their CCA estimations. Article Summary Strengths and limitations of this study This protocol explores the assumptions underlying the overlap assessment in overviews of systematic reviews, that so far have not been explicitly addressed. These assumptions include scope adjustments, publication-thread adjustments, structural missingness adjustments, and analysis at an overview or outcome level. We provide clear definitions for key overlap concepts that will guide authors for making their overlap assessments more explicit when using a matrix of evidence or corrected covered area approach. We plan to conduct exploratory analyses under different assumptions in a purposive sample of overview, hence, we will not comprehensively include all the overviews in the study period. We will conduct all the analysis calculating the corrected covered area for the whole matrices (overall approach) and for every possible pair of systematic reviews within each matrix (pairwise approach).
Introducción A pesar de la existencia de repositorios de registros de protocolos de revisiones sistemáticas (RSs) y de los esfuerzos por introducir conceptos como “investigación basada en la evidencia” para reducir la publicación de estudios sin valor, el número de RSs publicadas en los últimos años ha aumentado de forma constante, lo que pudo haberse exacerbado durante la pandemia del COVID-19. En este contexto las overviews (revisión de RSs) pueden ser una herramienta útil para que los profesionales de la salud se mantengan actualizados. Sin embargo, es frecuente encontrar RSs que responden a la misma pregunta de investigación. Uno de los aspectos metodológicos más conflictivos a la hora de realizar una overview es el solapamiento de los estudios primarios incluidos en las RSs. Objetivos Los objetivos de esta meta-investigación fueron describir la frecuencia del uso de estrategias para manejar el solapamiento de estudios primarios en overviews de intervenciones relacionadas con el ejercicio, las estrategias utilizadas para el manejo del solapamiento, y el reconocimiento por parte de los autores como limitación metodológica el no utilizar ninguna estrategia en el manejo del solapamiento. Método Se realizó una búsqueda sistemática con un enfoque sensible en MEDLINE (Ovid), Embase (Ovid), Cochrane Library, Epistemonikos y otros recursos de búsqueda hasta junio de 2022. Se incluyeron overviews de SRs que consideraron estudios primarios con cualquier diseño metodológico y evaluaron la efectividad de las intervenciones relacionadas con el ejercicio en cualquier condición de salud. La selección de las overviews se realizó por duplicado, y la extracción de los datos fue realizada por un investigador y comprobada por un segundo revisor. Se utilizó un formulario estándar de extracción de datos que consideró las características bibliométricas de las overviews, los aspectos metodológicos de las overviews, los resultados reportados por las overviews, y las estrategias utilizadas por los autores de las overviews para visualizar, cuantificar y resolver el solapamiento de los estudios primarios. Las características de las overviews incluidas se presentaron mediante estadística descriptiva y en forma narrativa utilizando tablas y figuras. Resultados principales Trescientos cincuenta y tres overviews publicadas entre 2005 y 2022 cumplieron los criterios de elegibilidad de esta meta-investigación. El objetivo principal más común de las overviews fue determinar la efectividad de diferentes intervenciones en una misma población (56,09%). Ciento sesenta y cuatro (46,46%) overviews utilizaron alguna estrategia para visualizar, cuantificar o resolver el solapamiento, siendo la matriz (86,49%), la frecuencia absoluta (47,89%) y los algoritmos diseñados por los propios autores (22,22%) los métodos más utilizados, respectivamente. Los autores de 16 (8,47%) de las overviews en los que no se reportó ninguna estrategia para manejar el solapamiento reconocieron este hecho como una limitación de su estudio. Conclusiones Menos de la mitad de los autores de las overviews de intervenciones relacionadas con el ejercicio utilizaron alguna estrategia para visualizar, cuantificar o resolver el solapamiento de los estudios primarios. Los métodos más utilizados tenían como objetivo resolver el solapamiento; sin embargo, sólo la mitad de los resúmenes lograron este objetivo. Una fracción baja de autores que no utilizaron alguna estrategia para manejar el solapamiento reconoció esto como una limitación. Futuras overviews deberían evaluar y manejar el solapamiento para conseguir resultados más válidos.
Abstract Introduction: An increasing growth of systematic reviews (SRs) presents notable challenges for decision-makers seeking to answer clinical questions. In 1997, an algorithm was created by Jadad to assess discordance in results across SRs on the same question. Our study aims to (a) replicate assessments done in a sample of studies using the Jadad algorithm to determine if the same SR would have been chosen, (b) evaluate the Jadad algorithm in terms of utility, efficiency, and comprehensiveness, and (c) describe how authors address discordance in results across multiple SRs. Methods and Analysis: We will use a database of 1218 overviews (2000-2020) created from a bibliometric study as the basis of our search for studies assessing discordance (called Discordant Reviews). This bibliometric study searched MEDLINE (Ovid), Epistemonikos, and Cochrane Database of Systematic Reviews for overviews. We will include any study using Jadad (1997) or another method to assess discordance. The first 30 studies screened at the full-text stage by two independent reviewers will be included. We will replicate the authors’ Jadad assessments. We will compare our outcomes qualitatively and evaluate the differences between our Jadad assessment of discordance and the authors’ assessment. Ethics and Dissemination: No ethics approval was required as no human subjects were involved. In addition to publishing in an open-access journal, we will disseminate evidence summaries through formal and informal conferences, academic websites, and across social media platforms. This is the first study to comprehensively evaluate and replicate Jadad algorithm assessments of discordance across multiple SRs.
Abstract Introduction: Network meta-analyses (NMAs) have gained popularity and grown in number due to their ability to provide estimates of comparative effectiveness of multiple treatments for the same condition. The aim of this study is to conduct a methodological review to compile a preliminary list of concepts related to bias in NMAs. Methods and Analysis: We included papers that present items related to bias, reporting, or methodological quality, papers assessing the quality of NMAs, or methods papers. We searched MEDLINE, the Cochrane library, and unpublished literature (July 2020). We extracted items related to bias in NMAs. An item was excluded if it related to general systematic review quality or bias, and included in currently available tools such as ROBIS or AMSTAR 2. We reworded items, typically structured as questions, into concepts (i.e. general notions). Results: 181 articles were assessed at full text and 57 were included. Of these articles, 12 were tools, checklists or journal standards, 12 were guidance documents for NMAs, 26 were articles related to bias or methods, and 7 were papers that assessed the quality of NMAs. These studies yielded 99 items of which the majority related to general systematic review quality and biases and were therefore excluded. The items we included were reworded into 22 concepts specific to bias in NMAs. Conclusions: A list of 22 concepts were included. This list is not intended to be used to assess biases in NMAs, but to inform the development of items to be included in our tool.
Abstract Background Multiple ‘overviews of reviews’ conducted on the same topic (“overlapping overviews”) represent a waste of research resources, and can confuse clinicians who are required to choose among competing treatments. We aimed to assess the frequency and characteristics of overlapping overviews. Methods MEDLINE, Epistemonikos and Cochrane databases were searched for overviews that: synthesised reviews of health interventions and conducted systematic searches. Overlap in topic was defined as: duplication of PICO elements, not representing an update of a previous overview, and not a replication. We also categorized the overviews as broad or narrow in scope. Results Of 541 overviews identified (2000–2018), 172 (32%) overlapped across similar PICO. The overlapping overviews fell within 13 WHO ICD-10 medical classifications and 63 topics. The overviews may have overlapped partially or completely, such that a similar portion, major component(s), or complete representation of an overview was duplicated. 149/172 (87%) overlapping overviews were characterized as broad in scope. Most frequently, broad overviews had targeted populations for which multiple interventions were addressed (44%), or least frequently, they had a targeted intervention for multiple populations (17%). Conclusions One third of overviews overlapped in content with a majority covering broad topic areas, and fewer considering subsets of the evidence. A multiplicity of overviews on the same topic adds to the ongoing waste of research resources, time and effort across medical disciplines. This study and the database of 172 overlapping overviews can provide a guide to authors about which topics are covered, and gaps in the evidence for future analysis.
Abstract Introduction: An increasing growth of systematic reviews (SRs) presents notable challenges for decision-makers seeking to answer clinical questions. Overviews of systematic reviews aim to address these challenges by summarising results of SRs and making sense of potentially discrepant SR results and conclusions. In 1997, an algorithm was created by Jadad to assess discordance in results across SRs on the same topic. Since this tool pre-dates the advent of overviews, it has been inconsistently applied in this context. Our study aims to (a) replicate assessments done in a sample of overviews using the Jadad algorithm to determine if the same SR would have been chosen, (b) evaluate the Jadad algorithm in terms of utility, efficiency, and comprehensiveness, and (c) describe how overviews address discordance in results across multiple SRs. Methods and Analysis: We will use a database of 1218 overviews (2000-2020) created from a bibliometric study as the basis of our search for overviews assessing discordance. This bibliometric study searched MEDLINE (Ovid), Epistemonikos, and Cochrane Database for overviews. We will include any overviews using Jadad (1997) or another method to assess discordance. The first 30 overviews screened at the full-text stage by two independent reviewers will be included. We will replicate Jadad assessments in overviews. We will compare our outcomes qualitatively and evaluate the differences between our Jadad assessment of discordance and the overviews’ assessment. Ethics and Dissemination: No ethics approval was required as no human subjects were involved. In addition to publishing in an open-access journal, we will disseminate evidence summaries through formal and informal conferences, academic websites, and across social media platforms. This is the first study to comprehensively evaluate and replicate Jadad algorithm assessments of discordance in SRs.
Abstract Objective To investigate pharmaceutical or medical device industry funding of patient groups. Design Systematic review with meta-analysis. Data sources Ovid Medline, Embase, Web of Science, Scopus, and Google Scholar from inception to January 2018; reference lists of eligible studies and experts in the field. Eligibility criteria for selecting studies Observational studies including cross sectional, cohort, case-control, interrupted time series, and before-after studies of patient groups reporting at least one of the following outcomes: prevalence of industry funding; proportion of industry funded patient groups that disclosed information about this funding; and association between industry funding and organisational positions on health and policy issues. Studies were included irrespective of language or publication type. Review methods Reviewers carried out duplicate independent data extraction and assessment of study quality. An amended version of the checklist for prevalence studies developed by the Joanna Briggs Institute was used to assess study quality. A DerSimonian-Laird estimate of single proportions with Freeman-Tukey arcsine transformation was used for meta-analyses of prevalence. GRADE (Grading of Recommendations Assessment, Development, and Evaluation) was used to assess the quality of the evidence for each outcome. Results 26 cross sectional studies met the inclusion criteria. Of these, 15 studies estimated the prevalence of industry funding, which ranged from 20% (12/61) to 83% (86/104). Among patient organisations that received industry funding, 27% (175/642; 95% confidence interval 24% to 31%) disclosed this information on their websites. In submissions to consultations, two studies showed very different disclosure rates (0% and 91%), which appeared to reflect differences in the relevant government agency’s disclosure requirements. Prevalence estimates of organisational policies that govern corporate sponsorship ranged from 2% (2/125) to 64% (175/274). Four studies analysed the relationship between industry funding and organisational positions on a range of highly controversial issues. Industry funded groups generally supported sponsors’ interests. Conclusion In general, industry funding of patient groups seems to be common, with prevalence estimates ranging from 20% to 83%. Few patient groups have policies that govern corporate sponsorship. Transparency about corporate funding is also inadequate. Among the few studies that examined associations between industry funding and organisational positions, industry funded groups tended to have positions favourable to the sponsor. Patient groups have an important role in advocacy, education, and research, therefore strategies are needed to prevent biases that could favour the interests of sponsors above those of the public. Systematic review registration PROSPERO CRD42017079265.
ObjectiveThis meta-research study aimed to describe the frequency of strategies to manage the overlap in overviews of exercise-related interventions.Study Design and SettingA systematic search in MEDLINE (Ovid), Embase (Ovid), Cochrane Library, Epistemonikos, and other sources was conducted from inception to June 2022. We included overviews of SRs that considered primary studies and evaluated the effectiveness of exercise-related interventions for any health condition. The overviews were screened by two authors independently, and the extraction was performed by one author and checked by a second. We found 353 overviews published between 2005 and 2022 met the inclusion criteria.ResultsOne hundred and sixty-four overviews (46%) used at least one strategy to visualise, quantify, or resolve overlap, with a matrix (32/164; 20%), absolute frequency (34/164; 21%), and authors' algorithms (24/164; 15%) being the most used methods, respectively. Of the 108 overviews that used some strategy to resolve the overlap, 79 (73%) succeeded. In overviews where no strategies to manage overlap were reported (n = 189/353; 54%), 16 overview authors (8%) recognised this as a study limitation.ConclusionLess than half of the authors of the overviews of exercise-related interventions used a strategy to visualise, quantify, or resolve overlap in the primary studies’ data. In the future, authors should report such strategies to communicate more valid results.
