Evolving Software to be ML-Driven Utilizing Real-World A/B Testing: Experiences, Insights, Challenges

ML-driven software is heralded as the next major advancement in software engineering; existing software today can benefit from being evolved to be ML-driven. In this paper, we contribute practical knowledge about evolving software to be ML-driven, utilizing real-world A/B testing. We draw on experiences evolving two software features from the Windows operating system to be ML-driven, with more than ten real- world A/B tests on millions of PCs over more than two years. We discuss practical reasons for using A/B testing to engineer ML-driven software, insights for success, as well as on-going real- world challenges. This knowledge may help practitioners, as well as help direct future research and innovations.