Assessing the robustness of meta-analytic results in information systems: publication bias and outliers

AbstractMeta-analytic studies serve to generate cumulative knowledge and guide evidence-based practice. However, publication bias and outliers threaten the accuracy and robustness of meta-analytic results. Unfortunately, most meta-analytic studies in information systems (IS) research do not assess the presence of these phenomena. Furthermore, some methods commonly used for the detection of publication bias are now recognised as inappropriate. We conduct a comprehensive assessment of four previously published meta-analytic studies in IS. We use multiple methods to assess the effects of publication bias and outliers on the meta-analytic results. Our findings indicate that publication bias and/or outliers have affected the results of three of the four meta-analytic studies. Some methods indicate that select meta-analytic means were misestimated by potentially more than 100%. Our analyses offer methodological exemplars that can be followed to assess the potential adverse effects of publication bias and outlie...