Better reporting for better research: a checklist for reproducibility

How easy is it to reproduce or replicate the findings of a published paper? In 2013 one researcher, Phil Bourne, asked just this. How easy would it be to reproduce the results of a computational biology paper? [1]. The answer: 280 hours. Such a number is surprising, given the theoretical reproducibility of computational research and given Bourne was attempting to reproduce work done in his own lab. Now at the National Institutes of Health (NIH) as Associate Director of Data Sciences, Bourne is concerned with the reproducibility of all NIH funded work, not just his own—and the problem is large. In addition to work in computational biology (which theoretically should be more easily reproducible than “wet lab” work), hallmark papers in cancer through to psychology have been flagged as largely unreproducible [2, 3]. Closer to home, GigaScience has carried out similar work to quantify reproducibility in their content. Despite being scrutinized and tested by seven referees, it still took about half a man-month worth of resources to reproduce the results reported in just one of the tables [4]. “Reproducibility” is now increasingly on the radar of funders and is making its rounds in the wider media as well, with concerns of reproducibility making headlines at The Economist [5] and New York Times [6], amongst other outlets.