Important Experimentation Characteristics: An Expert Survey

Background: Recent empirical studies indicate that online controlled experimentation is rarely systematically applied. A structured and complete experiment definition is the basis for systematic and trustworthy experimentation. Aims: As the first step towards guidelines for the definition of experiments, we explore experimentation application types that are conducted in practice. Additionally, we identify experiment definition characteristics that experts regard to considerable contribute to trustworthy experimentation. Method: An expert survey among fifteen industrial experts that published peer-reviewed publications was conducted. Results: In total, we identified fourteen types of applications with 32 concrete applications among the answers of the experts. The most frequently mentioned characteristics regarding the accuracy of an experiment were: success metrics, hypothesis, data quality metrics, guardrail metrics, alerting & shutdown, sizing, and segmentation. Conclusions: There are various applications for experimentation besides the ones mentioned in the literature. Most experts consider only about half of the known characteristics as relevant for the accuracy of an experiment.