Avoiding Decision-Making by Chance: Protecting Effect Size Estimates

Abstract Cohen's popular book titled Statistical Power Analysis for the Behavioral Sciences, coupled with recent challenges to statistical significance, has made "effect size" one of the hottest methodological topics of our time. Cohen recommends specific levels of effect size for "small," "medium," and "large" effects. However, even Cohen acknowledged these values are relative to the specific content and method in a given research situation. The purpose of this study is to determine the probabilities of attaining varying magnitudes of standardized effect sizes by chance and when protected by a .05-level statistical test. Monte Carlo procedures were used to generate standardized effect sizes in a one-way ANOVA situation with 2 through 5, 6, 8, and 10 groups having selected sample sizes from 5 to 500. Within each of the 91 number of group and sample size configurations, 100,000 replications were generated from a distribution of normal deviates. For each data set, the effect size was computed along with a statistical test of hypothesis at the .05 level. For each n/k combination, the proportion of effect sizes exceeding 0.1 to 2.0 in increments of .1 was computed for all cases and for those cases where the no difference hypothesis was rejected. There are trends that are common across all configurations. As the magnitude of effect size increases, the probability of getting such a difference by chance decreases as would be expected. Within a given number of samples situation, as sample size increases, as expected, the probability of getting such a difference by chance decreases. Within a given sample size, as the number of groups increases the probability of getting such a difference by chance increases. Another finding which is consistent across all configurations is that the significance test protected effect size probability is always equal to or less than the unprotected probability, in some cases dramatically so. It is clear that the addition of the significance test reduces the probability of finding a seemingly large effect size by chance. Such a protected effect size indicator could be an answer to the arguments posed by both those who protest against the use of the significance test and those who propose its use in judging the magnitude of an observed effect.