Evaluating T-wise testing strategies in a community-wide dataset of configurable software systems

Abstract Configurable software systems allow developers to maintain a unique platform and address a diversity of deployment contexts and usages. Testing configurable systems is essential because configurations that fail may potentially hurt users and degrade the project reputation. As extensively testing all valid configurations is infeasible in practice, several testing strategies have been proposed to recommend an optimal sample of configurations able to find most existing faults. However, up to now, we could not find studies comparing testing strategies with a community-wide dataset. Aiming at (i) comparing sampling testing strategies and (ii) understanding the location of faults, we use a community-wide dataset from the literature and compare suggested configurations from variations of five t-wise testing strategies (e.g., ICPL-T2, Chvatal-T4, and IncLing-T2). This comparison aims to find which strategies are faster, more comprehensive, effective on identifying faults, time-efficient, and coverage-efficient in this community-wide dataset and the reasons why a strategy fared better in one investigated property. Complementary, we investigate the dispersion of faults over classes and features from the dataset. As a result, we found that the dispersion of faults are usually concentrated in a few classes and features. Furthermore, fault-prone classes and features are distinguishable from classes and features safe of faults. Overall, we believe that with our results practitioners acquire the necessary knowledge to choose a testing strategy that best fits their needs. Moreover, researchers and tool builders are served with a bunch of opportunities to improve existing testing strategies and tools. For instance, they may incorporate information from fault-prone classes and features when selecting configurations to be tested in their testing strategies.