‘Good’ evidence for improved policy making: from hierarchies to appropriateness

Within the field of public health, and increasingly across other areas of social policy, there are widespread calls to increase or improve the use of evidence for policy making. Often these calls rest on an assumption that improved evidence utilisation will be a more efficient or effective means of achieving social goals. Yet, a clear elucidation of what can be considered ‘good evidence’ for policy use is rarely articulated. Many of the current discussions of best practice in the health policy sector derive from the evidence-based medicine (EBM) movement, embracing the ‘hierarchy of evidence’ in framing the selection of evidence – a hierarchy that places experimental trials as preeminent in terms of methodological quality. However, there are a number of difficulties associated with applying EBM methods of grading evidence onto policy making. Numerous public health authors have noted that the hierarchy of evidence is a judgement of quality specifically developed for measuring intervention effectiveness, and as such it cannot address other important health policy considerations such as affordability, salience, or public acceptability (Petticrew and Roberts, 2003). Social scientists and philosophers of knowledge have illustrated other problems in the direct application of the hierarchy of evidence to guide policy. Complex or structural interventions are often not conducive to experimental methods, and as such, a focus on evidence derived from randomised trials may shift policy attention away from broader structural issues (such as addressing the social determinants of health (Solar and Irwin, 2007)), to disease treatment or single element interventions. Social and behavioural interventions also present external validity problems to experimental methods and meta-analyses, as the mechanisms by which an intervention works in one social context may be very different or produce different results elsewhere (Cartwright, 2011). In these cases, policy makers may be better advised to look for evidence about the mechanism of effect, and evidence of local contextual features (Pawson et al., 2005). We argue that rather than adhering to a single hierarchy of evidence to judge what constitutes ‘good’ evidence for policy, it is more useful to examine evidence through the lens of appropriateness. It is important to utilise evidence to improve policy outcomes, yet the form of that evidence should vary depending on the multiple decision criteria at stake. Policy makers must therefore start by articulating their decision criteria in relation to a given problem or policy, so that the appropriate forms of evidence can be drawn on – from both epidemiological and clinical experiments (e.g. for questions of treatment effect), as well as from social scientific, social epidemiological, and multidisciplinary sources (e.g. for questions of complex causality, acceptability, human rights, etc.). Following this selection of types of evidence on the basis of appropriateness, the rigour and quality of the research can be assessed according to the evidentiary best practice standards of the discipline within which the evidence was produced. This approach speaks to calls to improve the use of evidence through ensuring rigour and methodological quality, yet recognises that good evidence is dictated by specific public health or social policy goals.