Same Data, Different Conclusions: Analysis of the New Zealand Drink-Driving Campaign Data

Everybody has an opinion on whether the New Zealand road safety television advertisements work to persuade people to behave better and thus reduce the road toll. The TV viewers disagree in conversation, and the experts disagree with one another in written reports, but the public wants to know whether the money is well spent and they can't understand why the experts can't tell them. This paper explains, in a language that is accessible to everyone, why we are so bad at monitoring the effects of advertising campaigns such as the road safety campaign. The paper explains how different results can be obtained from the same set of data and suggests that there is no objective way of judging between the different results. Moreover, the paper makes the claim that no amount of expertise can achieve a reliable result. The paper claims that the problems are inherent in the way the data are generated and collected, but a makes a controversial suggestion for a change to the way we view social experiments, so that the data generated is amenable to reliable analysis.