Robustness and External Validity: What do we Learn from Repeated Study Designs over Time?

The replication of studies is foundational to the scientific method. But replication can have different meanings varying from results verification, to reproduction, to reanalysis with an alternative specification. Few studies are repeated in the same setting. Yet this may be key when environmental or other mediating factors change over time, as it may help establish validity over time or shed light on why impact changes over time. One example is the varied impact a health intervention can have, particularly when targeting infectious diseases, as disease incidence may vary across years. This paper presents the findings of a study that repeats the same health intervention in the same site in three different years, estimating the effect of malaria testing and treatment on agricultural worker earnings, labor supply and productivity. We find a significant impact on worker earnings across the years, but the impact size varies over time. The treatment on the treated estimates are lower in a year when the malaria prevalence rate is low. The ‘treatment on the medically untreated’, which captures an information and behavioral effect identified in an earlier study, is smaller in years when prevalence is lower and the possibility of substituting into and out of lower effort, lower return tasks is absent. These results underline the importance of changes in the prevalence rate as well as the worker’s labor constraints. The results demonstrate that repetition, apart from providing a useful tool for validation, can also help shed light on reasons why effects may vary over time.