Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing

Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests. Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions.Learn how toUse the scientific method to evaluate hypotheses using controlled experiments Define key metrics and ideally an Overall Evaluation CriterionTest for trustworthiness of the results and alert experimenters to violated assumptionsBuild a scalable platform that lowers the marginal cost of experiments close to zeroAvoid pitfalls like carryover effects and Twyman's lawUnderstand how statistical issues play out in practice.