Debugging and maintaining pragmatically reused test suites

Abstract Context Pragmatic software reuse is a common activity in industry, involving the reuse of software artifacts not designed to anticipate that reuse. Objective There are two key issues in such tasks that have not been previously explored. (1) Subtle bugs can be inserted due to mistakes on the part of a developer performing the pragmatic reuse. The reused code, integrated in the target system, should be (re-)validated there. But it is not clear what validation strategies would be employed by professional developers, and which of these strategies would be most effective to detect and to repair these inserted bugs. (2) Although semi-automated reuse of the associated test suite has been previously proposed as a strategy to detect such inserted bugs, it is unknown if the reused test suite would be maintainable in practice and how its maintenance characteristics would compare against alternative strategies. Method We present two empirical studies with industrial developers to address these open issues. Results We find that industrial developers use a few strategies including test suite reuse, but that test suite reuse is more reliably effective at discovering and repairing bugs inserted during pragmatic reuse. We also find that, in general, semi-automatically reused test suites are slightly more maintainable than manually reused test suites, in pragmatic reuse scenarios; specific situations can vary wildly however. Participants suggested specific extensions to tool support for semi-automated reuse of test suites. Conclusions While various validation strategies are employed by industrial developers in the context of pragmatic reuse, none is as reliable and effective as test case reuse at discovering and repairing bugs inserted during pragmatic reuse. Despite the fact that semi-automatically reused test cases contain non-trivial adaptive code, their maintainability is equivalent to or exceeds that of manually reused test suites. The approach could be improved, however, by adopting the suggestions of our participants to increase usability.