60 Are manufacturers’ product claims supported by clinical evidence? an objective systematic evaluation of coronary stent product claims

Objectives With new product releases, manufacturers commonly use claims to promote the benefits of their products. In this case study, we take a systematic objective approach to evaluating the validity of these claims for the coronary stent market. By testing the claims found on manufacturers’ official websites against the published clinical evidence on each stent reviewed, we take an evidence-based approach to assess whether the claims are, in fact, supported by clinical outcomes. Can manufacturers’ claims be trusted in helping clinicians make informed choices on products, or must health systems beware and perform their own evidence-based due-diligence to make the most appropriate product formulary decisions? Method Coronary stents introduced and approved by the FDA from 2014 to present (January 2018) were identified for analysis. The product claims for each were determined from the respective manufacturers’ official websites. These claims were then investigated by researching relevant published papers in medical databases and search engines including PubMed, Google Scholar, ResearchGate, and Cochrane Library. With the support of a machine-assisted tool called EvidenceEngine, each paper was classified with an Evidence Quality score and Evidence Direction score. The Evidence Quality is determined based on the paper’s study design, population size, publication date, peer-review status, and potential for conflict of interest. The Evidence Direction is determined based on a sentiment analysis of the study’s conclusion relative to the claim being tested. The overall results are compiled using the Evidence Quality as a weighting factor. Results Thirty (30) claims were identified for the five (5) coronary stents released between January 2014 and January 2018. The search revealed 67 unique studies relevant to the claims. A potential conflict of interest between the author and the manufacturer (either direct or indirect) was declared in about half of these studies (37). Of the 30 claims, 50% (15) are supported by published evidence, with 26.67% (8) supported by clinical studies and 23.33% (7) by non-clinical studies. For 23.33% (7) of the claims, the evidence actually refutes the claim. For the remaining 26.67% (8) claims, no relevant published data could be identified, with five of these claims being too general or imprecise to be verifiable. Conclusions The results of the comparative analysis conclude that manufacturers’ claims about their products cannot be accepted at face value. An evidence-based approach is necessary to properly evaluate the validity of such claims. In the coronary stent market for new products introduced since 2014, only half of the manufacturers’ claims were found to be sufficiently supported by the published evidence. Approximately a quarter of these claims actually are refuted by the clinical evidence. With the use of a systematic approach to evaluating the claims based on evidence, health systems can make more informed and clinically sound decisions regarding medical device utilisation.