Theory is in the Eye of the Beholder: Exploring Difficulties with Validating Intervention Mechanisms: Theory is in the Eye of the Beholder

Behavior change researchers frequently base their interventions on theory, targeting specific mechanisms of change to help users achieve their goals. Thus HCI researchers have sought to examine whether their intervention impacts the intended mechanism of change, in addition to evaluating the overall effect of the intervention. Yet an open question remains: how do we know our interventions successfully target the mechanisms we intend? We present results from two validation studies and one user study showing the difficulties of ensuring that behavior change interventions target the mechanisms designers intend. Our findings indicate that experts disagree about what mechanism an intervention targets, that expert consensus on this matter can be hard to achieve, and that end users’ reflections indicate they may follow different mechanisms than those predicted by experts. We recommend that researchers collect data about multiple potential mechanisms that their intervention could operate through, rather than the common single-mechanism approach.