Comparing non-adequate test suites using coverage criteria
116
Citation
56
Reference
10
Related Paper
Citation Trend
Abstract:
A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is C-adequate.Keywords:
Test suite
Code coverage
Software testing
Test suite
Code coverage
Model-Based Testing
Completeness (order theory)
System under test
Model Transformation
Test Management Approach
Cite
Citations (17)
Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. In previous work, we studied two approaches to augmentation, one using a concolic test case generation algorithm and one using a genetic test case generation algorithm. We found that these two approaches behaved quite differently in terms of their costs and their abilities to generate effective test cases for evolving programs. In this paper, we present a hybrid test suite augmentation technique that combines these two test case generation algorithms. We report the results of an empirical study that shows that this hybrid technique can be effective, but with varying degrees of costs, and we analyze our results further to provide suggestions for reducing costs.
Test suite
Code coverage
Regression testing
Model-Based Testing
Code (set theory)
Test Management Approach
Cite
Citations (28)
As software evolves, engineers use regression testing to evaluate its fitness for release. Such testing typically begins with existing test cases, and many techniques have been proposed for reusing these cost-effectively. After reusing test cases, however, it is also important to consider code or behavior that has not been exercised by existing test cases and generate new test cases to validate these. This process is known as test suite augmentation. In this paper we present a directed test suite augmentation technique, that utilizes results from reuse of existing test cases together with an incremental concolic testing algorithm to augment test suites so that they are coverage-adequate for a modified program. We present results of an empirical study examining the effectiveness of our approach.
Test suite
Regression testing
Model-Based Testing
Code coverage
Test Management Approach
Test harness
System under test
Code (set theory)
Cite
Citations (48)
Test suite
Code coverage
Test Management Approach
Model-Based Testing
Regression testing
Test harness
Cost reduction
Cite
Citations (0)
Software testing plays a very important role in the software development process. Automated test generation tools increase the effectiveness and efficiency of software testing, and alleviate the problem of low efficiency caused by writing hand-crafted test cases. However, different test case generation methods vary in the size, code coverage, and fault detection capacity of the automatically-produced test suites. Automated test case generation tool based on random testing, Randoop as a representative, randomly and incrementally generates a large number of method sequences, which gives various possible combinations of calling methods, but the size of the test suite is not proportional to test quality. Therefore, there exists a lot of redundancy in the test cases. This paper proposes Randoop-TSR, an approach for identifying and eliminating redundant test cases on the basis of Randoop to improve the process of test generation. Our approach adopts three strategies to realize the removal of redundancy, namely: (i) similarity-based input sequence selection; (ii) redundant and duplicate assert statements elimination based on test smell detection; (iii) redundant test cases elimination without breaking test requirements (i.e., code coverage and mutation score). Randoop-TSR can eliminate redundancy effectively, and greatly reduce the size of test suites and execution time. Furthermore, our approach improves the efficiency and understandability of test cases while retaining code coverage and mutation score.
Test suite
Code coverage
Random testing
Fuzz testing
Test Management Approach
Test compression
System under test
Model-Based Testing
Cite
Citations (4)
Test suite
Code coverage
Model-Based Testing
Test Management Approach
Regression testing
Test harness
Manual testing
Cite
Citations (11)
As a software system evolves, its test suite can accumulate redundancies over time. Test minimization aims at removing redundant test cases. However, current techniques remove whole test cases from the test suite using test adequacy criteria, such as code coverage. This has two limitations, namely (1) by removing a whole test case the corresponding test assertions are also lost, which can inhibit test suite effectiveness, (2) the issue of partly redundant test cases, i.e., tests with redundant test statements, is ignored. We propose a novel approach for fine-grained test case minimization. Our analysis is based on the inference of a test suite model that enables automated test reorganization within test cases. It enables removing redundancies at the test statement level, while preserving the coverage and test assertions of the test suite. We evaluated our approach, implemented in a tool called Testler, on the test suites of 15 open source projects. Our analysis shows that over 4,639 (24%) of the tests in these test suites are partly redundant, with over 11,819 redundant test statements in total. Our results show that Testler removes 43% of the redundant test statements, reducing the number of partly redundant tests by 52%. As a result, test suite execution time is reduced by up to 37% (20% on average), while maintaining the original statement coverage, branch coverage, test assertions, and fault detection capability.
Test suite
Code coverage
Test Management Approach
Minification
Test harness
Test script
System under test
Model-Based Testing
Cite
Citations (27)
For developing successful software, testing is a very important component. In software testing, providing input, executes it and check expected output. Many techniques which automatically produce inputs have been proposed over the years, and today are able to produce test suites with high code coverage. In software testing a common scenario is that test data are generated and a tester manually adds test cases. It is a difficult task to generate test cases manually but it is important to produce small representative test sets and this representativeness is typically measured using code coverage. But there is a fundamental problem with the common approach of targeting one coverage goal at a time. Coverage goals are not independent, not equally difficult, and sometimes infeasible—the result of test generation is therefore dependent on the order of coverage goals and how many of them are feasible. For solving these problems, propose a novel paradigm which is generation of whole test suite based on search based testing. Instead of evolving each test case individually, evolve all the test cases in a test suite at the same time. At the end, the best resulting test suite is minimized.
Test suite
Code coverage
Test Management Approach
Model-Based Testing
Regression testing
Test harness
Cite
Citations (0)
Automated test case generation tools help businesses to write tests and increase the safety net provided by high regression test coverage when making code changes. Test generation needs to cover as much as possible of the uncovered code while avoiding generating redundant tests for code that is already covered by an existing test-suite. In this paper we present our work on a tool for the real world application of integrating formal analysis with automatic test case generation. The test case generation is based on coverage analysis using the Java bounded model checker (JBMC). Counterexamples of the model checker can be translated into Java method calls with specific parameters. In order to avoid the generation of redundant tests, it is necessary to measure the coverage in the exact same way as JBMC generates its coverage goals. Each existing coverage measurement tool uses a slightly different instrumentation and thus a different coverage criterion. This makes integration with a test case generator based on formal analysis difficult. Therefore, we developed BlueCov as a specific runtime coverage measurement tool which uses the exact same coverage criteria as JBMC does. This approach also allows for incremental test-case generation, only generating test coverage for previously untested code, e.g., to complete existing test suites.
Code coverage
Test suite
Test Management Approach
Regression testing
White-box testing
Code (set theory)
Model-Based Testing
Test harness
Cite
Citations (1)
Test suite reduction techniques decrease the cost of software testing by removing the redundant test cases from the test suite while still producing a reduced set of tests that yields the same level of code coverage as the original suite. Most of the existing approaches to reduction aim to decrease the size of the test suite. Yet, the difference in the execution cost of the tests is often significant and it may be costly to use a test suite consisting of a few long-running test cases. Thus, this paper proposes an algorithm, based on the concept of test irreplaceability, which creates a reduced test suite with a decreased execution cost. Leveraging widely used benchmark programs, the empirical study shows that, in comparison to existing techniques, the presented algorithm is the most effective at reducing the cost of running a test suite.
Test suite
Regression testing
Benchmark (surveying)
Code coverage
Model-Based Testing
Test Management Approach
System under test
Cite
Citations (24)