A reproducibility and validity study of a systematic review on ischemic cardiopathy. The Study Group of the Quality of Care (GRECA)

BACKGROUND: The systematic literature reviews have been proposed as a method of scientific evidence identification since they protect the final product from the subjectivity of each primary source reviewer. However, it is not known whether the different ways of evidence synthesis accomplish suitable criteria of objectivity, reliability and biases protection so as to be considered scientifically valid. An experiment of reliability and validity of a systematic literature review about coronary heart disease was carried out. MATERIAL AND METHODS: Study of blind concordance between two independent reviewers for the identification, selection, retrieval and quality evaluation of the articles by using the same protocol. The concordance was analysed by the kappa index for two observers in different categories. The validity was evaluated throughout the acceptability of the review users. RESULTS: The concordance for their identification capacity was poor although they used the same key words (869 versus 476). But the concordance improved when considering selection (26.6% versus 29.2%), retrieval (agreement = 76%) classification by kind of article (kappa = 0.60) and scoring by strength of the evidence (kappa = 0.87). The acceptability was high among review users. CONCLUSIONS: It would be assumed that, even under tight rules of performance, the systematic literature reviews are not completely protected against some biases which could damage their validity in a non easily controllable form. The implication of reviewers, experts in documentation and users of the literature, together with pilot studies performed previous to the review, seems to be the best way to yield better results.