Standing on the shoulders of bias: lack of transparency and reporting of critical rigor characteristics in pain research
Ashley N. PlumbJoseph B. LesnakGiovanni BerardiKazuhiro HayashiAdam J. JanowskiAngela F. SmithDana BaileyCassie KerkmanZoe E. KienenbergerBen R. MartinEmily S. PattersonHannah Van RoekelCarol VanceKathleen A. Sluka
10
Citation
46
Reference
10
Related Paper
Citation Trend
Abstract:
Abstract Rigorous experimental design with transparent reporting in biomedical science reduces risk of bias and allows for scientists to judge the quality of the research. Basic factors of rigor such as blinding, randomization, power analysis, and inclusion of both sexes impact the reproducibility by reducing experimental bias. We designed a systematic study to analyze basic factors of rigor, inclusion of sex, and whether data were analyzed or disaggregated by sex over the past 10 years in the journal PAIN . Studies that included humans reported randomization in 81%, blinding in 48%, and the use of a power analysis calculation in 27% over the past 10 years. Studies that included mice reported randomization in 35%, blinding in 70%, and the use of a power analysis in 9%. Studies that included rats reported randomization in 38%, blinding in 63%, and the use of power analysis in 12%. This study also found that human studies consistently included both sexes over the past decade, but less than 20% of data were disaggregated or analyzed for sex differences. Although mouse and rat studies predominately used males only, there has been a slight increase in inclusion of both sexes over the past few years. Justification for single-sex studies was below 50% in both human and rodent data. In both human and animal studies, transparency in reporting of experimental design and inclusion of both sexes should be considered standard practice and will result in improved quality and reproducibility of published research.Keywords:
Blinding
Clinical study design
Research Design
Reporting bias
Rigour
Introduction: Randomized Controlled Trial (RCTs) is the gold standard to objectively assess the effect of a treatment. The RCTs methodology must be particularly rigorous to achieve strong evidence of efficiency. To help improve the quality of RCTs, a group of experts established a list of recommendations, adopted by most international journals, called the CONSORT (CONsolidated Standards of Reporting Trials) Statement. First published in 1996, it was actualised in 2001 and 2010. In this study, we assessed the implementation of the CONSORT Statement criteria in the field of adult high grade gliomas. We also aimed to identify criteria associated with higher quality of RCTs. METHODS: We searched PUBMED to retrieve all RCTs concerning high grade gliomas published between the 1st January 1990 and the 1st March 2016. The quality of these RCTs was assessed by completing a modified CONSORT Score containing 33 items. This work was done independently by two investigators and every discordance was resolved by consensus. We also extracted data that seemed relevant to assess the quality of RCTs. Results: Eighty-four published RCTs were identified. The median CONSORT Score was 19 (range: 3-30). Items were not equally reported and items regarding the method of randomization, the blinding or the accessibility of the protocol were reported in less than 25% of RCTs which could raise important biases and led to inappropriate interpretation of the results. However, the CONSORT Score constantly improved over the years. Before the onset of the CONSORT Statement in 1996, the median CONSORT Score was 13 (range: 4-19) whereas it was equal to 18 (range: 3-26) for the period 1996-2004 and equal to 22 (range: 6-30) after 2005. A higher CONSORT Score was observed when RCTs were published in journal with impact factor above 10 (24 vs 17, p<0.001). RCTs that enrolled more than 200 patients had a significantly higher CONSORT Score (20.6 vs 16.4, p=0.004). CONCLUSION: This study demonstrated a continuous improvement over the years of the CONSORT Score of published RCTs in the field of high grade gliomas. We identified three factors associated with a better report of RCTs: the date of publication, the impact factor of the journal and the number of patients included in the study.
Blinding
Gold standard (test)
Quality Score
Cite
Citations (0)
Blinding
Cite
Citations (62)
Letters15 February 2011CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomized TrialsZhao-xiang Bian, MD, PhD and Hong-cai Shang, MD, PhDZhao-xiang Bian, MD, PhDFrom Hong Kong Baptist University, Kowloon Tong, Hong Kong, China, and Tianjin University of Traditional Chinese Medicine, Tianjin 300193, China. and Hong-cai Shang, MD, PhDFrom Hong Kong Baptist University, Kowloon Tong, Hong Kong, China, and Tianjin University of Traditional Chinese Medicine, Tianjin 300193, China.Author, Article, and Disclosure Informationhttps://doi.org/10.7326/0003-4819-154-4-201102150-00016 SectionsAboutFull TextPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail TO THE EDITOR:The CONSORT (Consolidated Standards of Reporting Trials) 2010 statement (1) no longer advocates reporting the success rate of blinding in trials. We believe that this decision is unwise and will seriously undermine the quality of trials. The success of blinding relates directly to the quality of a trial and directly affects the outcome of interventions.The importance of blinding is well known, and blinding helps to achieve quality assurance in trials. However, quality assurance does not rely on the setting but on the successful implementation of the trial protocol. The success rate of blinding is not only ...References1. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152:726-32. [PMID: 20335313] LinkGoogle Scholar2. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al; CONSORT GROUP (Consolidated Standards of Reporting Trials). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663-94. [PMID: 11304107] LinkGoogle Scholar3. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al; Consolidated Standards of Reporting Trials Group. CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63:1-37. [PMID: 20346624] CrossrefMedlineGoogle Scholar4. Freedman B. Equipoise and the ethics of clinical research. N Engl J Med. 1987;317:141-5. CrossrefMedlineGoogle Scholar5. Sackett DL. Turning a blind eye: why we don't test for blindness at the end of our trials [Letter]. BMJ. 2004;328:1136. [PMID: 15130997] CrossrefMedlineGoogle Scholar Author, Article, and Disclosure InformationAffiliations: From Hong Kong Baptist University, Kowloon Tong, Hong Kong, China, and Tianjin University of Traditional Chinese Medicine, Tianjin 300193, China.Disclosures: None disclosed. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoThe Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration Douglas G. Altman , Kenneth F. Schulz , David Moher , Matthias Egger , Frank Davidoff , Diana Elbourne , Peter C. Gøtzsche , Thomas Lang , and CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomized Trials Kenneth F. Schulz , Douglas G. Altman , David Moher , and CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomized Trials Kenneth F. Schulz , Douglas G. Altman , and David Moher Metrics Cited ByAcupuncture for post-operative cognitive dysfunction: a systematic review and meta-analysis of randomized controlled trialsSix-Dose Intravenous Tranexamic Acid Regimen Further Inhibits Postoperative Fibrinolysis and Reduces Hidden Blood Loss following Total Knee ArthroplastySitagliptin on carotid intima-media thickness in type 2 diabetes and hyperuricemia patients: a subgroup analysis of the PROLOGUE studyTranexamic acid attenuates inflammatory effect and modulates immune response in primary total knee arthroplasty: a randomized, placebo-controlled, pilot trialTranexamic acid safely reduces hidden blood loss in patients undergoing intertrochanteric fracture surgery: a randomized controlled trialFacilitating Existing Success and Continued Growth at AnnalsATS through Better Study ReportingThe cardiovascular risk of celecoxib for knee osteoarthritisAcupuncture and moxibustion for pain relief and quality of life improvement in patients with knee osteoarthritisTraditional Chinese exercise for COVID-19Using a machine learning algorithm to predict acute graft-versus-host disease following allogeneic transplantationExercise Rehabilitation Improves Cardiac Volumes and Functional Capacity in Patients With Endomyocardial FibrosisA nutraceutical diet based on Lespedeza spp ., Vaccinium macrocarpon and Taraxacum officinale improves spontaneous feline chronic kidney diseaseSocial marketing approaches to nutrition and physical activity interventions in early care and education centres: a systematic reviewAcupuncture for postprandial distress syndrome (APDS): study protocol for a randomized controlled trialTargeted LOWering of Central Blood Pressure in patients with hypertension: Baseline recruitment, rationale and design of a randomized controlled trial (The LOW CBP study)Effects in dogs with behavioural disorders of a commercial nutraceutical diet on stress and neuroendocrine parametersClinical evaluation of a nutraceutical diet as an adjuvant to pharmacological treatment in dogs affected by Keratoconjunctivitis siccaEffect of green tea consumption on blood pressure: A meta-analysis of 13 randomized controlled trialsThe impact of pancreaticojejunostomy versus pancreaticogastrostomy reconstruction on pancreatic fistula after pancreaticoduodenectomy: meta‐analysis of randomized controlled trialsCause-effect relationships in nutritional intervention studies for health claims substantiation: guidance for trial designAntiviral therapy after curative treatment of hepatitis B/C virus-related hepatocellular carcinoma: A systematic review of randomized trialsFuture Research in Acupuncture: Better Design and Analysis for Novel and Valid FindingsTacrolimus versus cyclosporin for simultaneous pancreas-kidney transplant recipientsPilot study of vitamin D supplementation in adults with cystic fibrosis pulmonary exacerbationWater-filtered infrared A for the treatment of chronic venous stasis ulcers of the lower legs at home: a randomized controlled blinded study 15 February 2011Volume 154, Issue 4Page: 290-291KeywordsClinical trialsConflicts of interestHealth care providersHealth care quality assessment ePublished: 15 February 2011 Issue Published: 15 February 2011 CopyrightCopyright © 2011 by American College of Physicians. All Rights Reserved.PDF DownloadLoading ...
Blinding
Cite
Citations (107)
Blinding
Trial registration
Quality Score
Research Design
Cite
Citations (57)
Abstract Rigorous experimental design with transparent reporting in biomedical science reduces risk of bias and allows for scientists to judge the quality of the research. Basic factors of rigor such as blinding, randomization, power analysis, and inclusion of both sexes impact the reproducibility by reducing experimental bias. We designed a systematic study to analyze basic factors of rigor, inclusion of sex, and whether data were analyzed or disaggregated by sex over the past 10 years in the journal PAIN . Studies that included humans reported randomization in 81%, blinding in 48%, and the use of a power analysis calculation in 27% over the past 10 years. Studies that included mice reported randomization in 35%, blinding in 70%, and the use of a power analysis in 9%. Studies that included rats reported randomization in 38%, blinding in 63%, and the use of power analysis in 12%. This study also found that human studies consistently included both sexes over the past decade, but less than 20% of data were disaggregated or analyzed for sex differences. Although mouse and rat studies predominately used males only, there has been a slight increase in inclusion of both sexes over the past few years. Justification for single-sex studies was below 50% in both human and rodent data. In both human and animal studies, transparency in reporting of experimental design and inclusion of both sexes should be considered standard practice and will result in improved quality and reproducibility of published research.
Blinding
Clinical study design
Research Design
Reporting bias
Rigour
Cite
Citations (10)
To assess whether clinical trials published in the orthodontic literature comply with the Consolidated Standards of Reporting Trials (CONSORT) statement, and whether compliance has changed over time.Retrospective study.Clinical trials published in four orthodontic journals (American Journal of Orthodontics and Dentofacial Orthopedics, Angle Orthodontist, European Journal of Orthodontics, and Journal of Orthodontics) at three time points [1995/6 (pre-CONSORT), 2000/1 (post-CONSORT) and 2005/6 (post revised-CONSORT)].Trials were scored using a 36-point checklist developed from the CONSORT statement. Each item on the checklist was scored as either included, not included, or not applicable if the trial design did not allow its inclusion. The proportion of trials describing randomization, allocation concealment, and blinding was also calculated. The mean scores were compared between time points and between journals.A total of 151 reports were included. The mean score was 41·5% (SD 10·3). The mean post-CONSORT score was higher than the pre-CONSORT score (P = 0·01; WMD, 5·73; 95% CI, 1·64-9·82), and the mean post revised-CONSORT was also higher than the pre-CONSORT score (P = 0·01; WMD, 4·44; 95% CI, 1·02-7·87). At the post revised-CONSORT time point, the mean score in the journals which had adopted the CONSORT statement was higher than for those which had not (P<0·001; chi square, 17·78; df = 3). Reporting of randomization, allocation concealment and blinding was inadequate at all three time points.The reporting of clinical trials in the orthodontic literature has improved since the publication of the CONSORT statement, particularly in journals which have adopted the statement. However, the reporting of randomization, allocation concealment and blinding remains inadequate.
Statement (logic)
Evidence-based dentistry
Cite
Citations (34)
Blinding
Trial registration
Cite
Citations (33)
Objective To evaluate the reporting quality of randomized controlled trials (RCTs) in seven military medical journals. Methods Seven journals in 2007,including Medical Journal of Chinese People's Liberation Army,Journal of South Medical University,Journal of Second Military Medical University,Journal of Third Military Medical University,Journal of Fourth Military Medical University,Bulletin of the Academy of Military of Medical Sciences and Academic Journal of PLA Postgraduate Medical School,were handsearched. We identified RCTs labeled and assessed the quality of these reports using the Consolidated Standards for Reporting of Trials (CONSORT) statement. Results We identified 99 RCTs,but found an incorrect randomized method was used in 6 RCTs. According to the items in the CONSORT statement in 93 RCTs,62 (66.7%) RCTs described baseline demographic and clinical characteristics in each group. Sixteen (17.2%) RCTs mentioned the method of random sequence generation,with 5 (5.4%) using a computer allocation. Only 1 RCT had adequate allocation concealment. Only 9 (9.7%) RCTs used blinding,with 2 mentioning blinding,1 using single blinding and 6 described as double-blind (2 were correct). Zero (0%) reported the sample size calculation and 1 RCT reported the intention-to-treat (ITT) analysis. Conclusion The reporting quality of RCTs in seven journals is poor. The CONSORT statement should be used to standardize the reporting of RCTs.
Blinding
Medical journal
Cite
Citations (0)
Background: Randomized Controlled Trials (RCTs) are reliable methods for the evaluation of treatment effectiveness, which should be rigorous and must report with clarity. This study aimed to assess the compliance of published RCTs about the effect of physical activity on primary dysmenorrhea with the CONSORT 2017 statement. Materials and Methods: In this study, the review of literature was carried out based on Consolidated Standards Of Reporting Trials (CONSORT). All the clinical trials focused on the effect of the physical activity on primary dysmenorrhea indexed in Web of Science, Pubmed, Scopus, Google Scholar, Science Direct, Embase, Magiran and Scientific Information Database (SID) were searched using keywords of dysmenorrhea, randomized clinical trial, physical activity and exercise from 2000 to 2019. Out of 1423 articles, 30 RCTs were critically appraised using CONSORT 2017 checklist. The reporting quality score of articles was identified between zero and 43. Results: The compliance rate with the CONSORT checklist was 55.58%. The mean (SD) score of the reporting quality was 23.37 (-5.15) with a minimum of 16 and a maximum of 37. The maximum weakness was in reporting the sample size and full trial protocol 23.33% and 6.67% respectively. Regarding new items of the consort 2017, if the blinding was not possible, the description of any attempts to limit bias was not described in 70% of articles. Conclusions: Reporting sample size, trial protocol, method of blinding, and control of bias are issues that require more attention in reporting of RCT studies. We recommend that the authors use the CONSORT 2017 statement for conducting and reporting the clinical trials.
Blinding
Critical appraisal
Cite
Citations (5)
To assure readers that study results are scientifically valid, the methods of a clinical trial should be described adequately. Since randomization, blinding, and intention-to-treat-analysis are major bias-reducing techniques, these aspects should be reported most accurately. The Consolidated standards of reporting trials (CONSORT) are recommendations to improve the reporting of trials. CONSORT requires that trial authors describe basic methodological aspects that readers need to appraise the strengths of reported clinical trials. This article presents the CONSORT recommendations and explains some of their main aspects. From now on, the Journal of Hand Surgery will use CONSORT to assist authors of randomized controlled trials in improving the description of their studies. We believe that this decision increases the scientific validity of study reports and helps readers when critically appraising articles.
Presentation (obstetrics)
Cite
Citations (19)