Stability of Product-Line Samplingin Continuous Integration.

Companies strive to implement continuous integration into their development process to ensure the quality of their systems. Regression testing within the CI process considers the efficient re-test of systems after changes. However, even with regression testing, it is not feasible to test all configurations from a highly-configurable software system due to the combinatorial-explosion problem. Numerous sampling algorithms have been proposed that aim at computing a considerably smaller yet sufficiently representative set of configurations to be tested. Those algorithms are typically evaluated with regard to efficiency (i.e., number of configurations in a sample and computational effort for generating a sample) and effectiveness (i.e., feature-interaction coverage or number of faults detected). In this paper, we argue that a further crucial characteristic of sampling algorithms is their tendency to produce similar configurations when applied consecutively to an evolving configurable system. We propose sampling stability as a new evaluation criterion for sampling algorithms. We present a procedure to compute the sampling stability of sampling algorithms based on the similarity between consecutive samples. In our evaluation, we compare the sampling stability of multiple established t-wise sampling algorithms on large real-world systems.