The problem of mission planning for multiple autonomous agents, including path planning and task scheduling, is often complex. Recent efforts aiming at solving this problem have explored ways of us ...
Autonomous systems are supposed to automatically plan their actions and execute the plan without human intervention. In this paper, we propose a model-based two-layer frame-work for policy synthesis and test-case generation for autonomous systems. At the high-level layer of the framework, we have two kinds of methods for synthesising policies whose correctness is guaranteed by model checking. The autonomous system's controller executes synthesised policies at the low-level layer. As the kinematics of autonomous systems is often nonlinear and the environment may influence the results of their actions, formally verifying the controllers is extremely difficult. We propose a novel method for generating test cases for the controllers at the low-level layer. The method employs reinforcement learning for test-case generation and model checking to ensure that the test cases faithfully realise the execution of the policy. The framework is designed in Uppaal Stratego, which integrates model checkers and algorithms for policy synthesis. Therefore, the framework separates concerns and seamlessly interchanges the information between two layers.
Many test prioritisation techniques have been proposed in order to improve test effectiveness of Continuous Integration (CI) pipelines. Particularly, diversity-based testing (DBT) has shown promising and competitive results to improve test effectiveness. However, the technical and practical challenges of introducing test prioritisation in CI pipelines are rarely discussed, thus hindering the applicability and adoption of those proposed techniques. This research builds on our prior work in which we evaluated diversity-based techniques in an industrial setting. This work investigates the factors that influence the adoption of DBT both in connection to improvements in test cost-effectiveness, as well as the process and human related challenges to transfer and use DBT prioritisation in CI pipelines. We report on a case study considering the CI pipeline of Axis Communications in Sweden. We performed a thematic analysis of a focus group interview with senior practitioners at the company to identify the challenges and perceived benefits of using test prioritisation in their test process. Our thematic analysis reveals a list of ten challenges and seven perceived effects of introducing test prioritisation in CI cycles. For instance, our participants emphasized the importance of introducing comprehensible and transparent techniques that instill trust in its users. Moreover, practitioners prefer techniques compatible with their current test infrastructure (e.g., test framework and environments) in order to reduce instrumentation efforts and avoid disrupting their current setup. In conclusion, we have identified tradeoffs between different test prioritisation techniques pertaining to the technical, process and human aspects of regression testing in CI. We summarize those findings in a list of seven advantages that refer to specific stakeholder interests and describe the effects of adopting DBT in CI pipelines.
This is the replication package for the analysis done in the paper "An Industrial Study on the Challenges and Effects of Diversity-based Testing in Continuous Integration". The package includes: (i) CSV files with data on test case and corresponding feature coverage, as well as test execution data; (ii) CSV files including failure coverage for the executed techniques; (iii) R scripts to re-run our visual and statistical analysis when comparing techniques results; and (iv) An R Markdown file (rendered into HTML) detailing the steps of our analysis with the coresponding code.
The data set used in the empirical exploration of RISKS reports. This data set acts as a basis for further research and investigation to categorize and analyze software-related fatal failures spanning more than 30 years.
A method for model-based test generation of safety-critical embedded applications using Programmable Logic Controllers and implemented in a programming language such as Function Block Diagram (FBD) is described. The FBD component model is based on the IEC 1131 standard and it is used primarily for embedded systems, in which timeliness is an important property to be tested. Our method involves the transformation of FBD programs with timed annotations into timed automata models which are used to automatically generate test suites. Specifically we demonstrate how to use model transformation for formalization and model-checking of FBD programs using the UPPAAL tool. Many benefits emerge from this method, including the ability to automatically generate test suites from a formal model in order to ensure compliance to strict quality requirements including unit testing and specific coverage measurements. The approach is experimentally assessed on a train control system in terms of consumed resources.
The emergence of new technologies in software testing has increased the automation and flexibility of the testing process. In this context, the adoption of agents in software testing remains an active research area in which various agent methodologies, architectures, and tools are employed to improve different test problems. Even though research that investigates agents in software testing has been growing, these agent-based techniques should be considered in a broader perspective. In order to provide a comprehensive overview of this research area, which we define as agent-based software testing (ABST), a systematic mapping study has been conducted. This mapping study aims to identify the topics studied within ABST, as well as examine the adopted research methodologies, identify the gaps in the current research and point to directions for future ABST research. Our results suggest that there is an interest in ABST after 1999 that resulted in the development of solutions using reactive, BDI, deliberative and cooperate agent architectures for software testing. In addition, most of the ABST approaches are designed using the JADE framework, have targeted the Java programming language, and are used at system-level testing for functional, non-functional and white-box testing. In regards to regression testing, our results indicate a research gap that could be addressed in future studies.
This is the replication package for the analysis done in the paper "An Industrial Study on the Challenges and Effects of Diversity-based Testing in Continuous Integration". The package includes: (i) CSV files with data on test case and corresponding feature coverage, as well as test execution data; (ii) CSV files including failure coverage for the executed techniques; (iii) R scripts to re-run our visual and statistical analysis when comparing techniques results; and (iv) An R Markdown file (rendered into HTML) detailing the steps of our analysis with the coresponding code.
Early design artifacts of embedded systems, such as architectural models, represent convenient abstractions for reasoning about a system's structure and functionality. One such example is the Electronic Architecture and Software Tools-Architecture Description Language (EAST-ADL), a domain-specific architectural language that targets the automotive industry. EAST-ADL is used to represent both hardware and software elements, as well as related extra-functional information (e.g., timing properties, triggering information, resource consumption). Testing architectural models is an important activity in engineering large-scale industrial systems, which sparks a growing research interest. The main contributions of this paper are: (i) an approach for creating energy-related mutants for EAST-ADL architectural models, (ii) a method for overcoming the equivalent mutant problem (i.e., the problem of finding a test case which can distinguish the observable behavior of a mutant from the original one), (iii) a test generation approach based on UPPAAL Statistical Model Checker (SMC), and (iv) a test selection criteria based on mutation analysis using our MATS tool.