On the proposal and evaluation of a test-enriched dataset for configurable systems

Configurable systems offer advantages compared to single systems since developers should maintain a unique platform to address a diversity of deployment contexts and usages. To ensure that all configurations correctly execute, developers spend considerable effort testing different system configurations. This testing process is essential because configurations that fail may potentially hurt user experience and degrade the reputation of a project. Previous studies have reported and created repositories of open-source configurable systems, although they neglected their test suites. Considering the importance of testing configurable systems, we reviewed the literature to find test suites of open-source configurable systems. As we found only 10 configurable systems with test suite available and considering that a test suite for configurable systems may be useful for different research topics, we created test suites for 20 additional configurable systems and evaluated the test suites coverage of all 30 configurable systems. Surprisingly, our test suites were able to find several failures in existing systems, mainly because of feature interactions, which enforces the need of test suites available for open source configurable systems. Aiming at finding common characteristics for fault-prone components (e.g., classes) on configurable systems, we group them based on software quality metrics (e.g., coupling between objects and lines of code). As result, we found that 44% of the configurable systems of our dataset have failures and these failures are concentrated in few classes.