Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology [Registered Report Stage 1 Protocol]

Abstract Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, there is evidence that such variation may far exceed what might be produced by sampling error. This evidence comes from a growing meta-research agenda that seeks to describe and explain variation in reliability of scientific results. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. The best evidence for this comes from a recent social science study that asked 29 different research teams to answer the same question independently by analyzing the same data set. Although many of the effect sizes were similar, some differed substantially from the average. We plan to implement an analogous study in ecology and evolutionary biology, a field in which there has been no empirical exploration of the variation in effect sizes or model predictions of dependent variables generated by analytical decisions of different researchers. We have obtained two unpublished data sets, one from evolutionary ecology and one from conservation ecology, and we will recruit as many independent scientists as possible to conduct analyses of these data to answer prespecified research questions. We will also recruit peer reviewers to rate the analyses based on their methodological descriptions so that we have multiple ratings of each analysis. Next we will quantify the variability in choices of independent variables among analyses and, using meta-analytic techniques, describe and quantify the degree of variability among effect sizes and predicted values for each of the data sets. Finally, we will quantify the extent to which deviation of individual effect sizes and predicted values from the meta-analytic mean for that data set is explained by peer review ratings and by the ‘uniqueness’ of the set of variables chosen for the analysis by each team.