When ANOVA gets it wrong: A re-introduction to the Regression Discontinuity design

When observations are not randomly assigned to conditions, assumptions of traditional estimation methods such as ANOVA or linear regression will be violated and can lead to biased and inconsistent estimators. However, there exist quasi-experimental designs that can be used to infer causality. One of these designs, the Regression Discontinuity (RD) design, allows for the drawing of proper causal conclusions when observations are not randomized to groupings, as long as the selection process is correctly modeled. A review of published articles shows that this design is mostly unknown to researchers in applied psychology and management. We show analytically where the bias in estimates originates when observations are assigned to conditions based on a quantifiable variable. We also use Monte-Carlo simulations to demonstrate how incorrect specification, based on the ANOVA model, can lead to biased estimates of the treatment effect. Because the RD design is easy to use and can be applied in many settings, we dis...