A quantitative comparison of the effects of modeling approaches on system verification using a controlled challenge problem

To reduce program risks, engineering methods capitalizing on modeling and machine assistance have been extensively investigated within systems engineering (and more specifically requirements engineering) literature over the past 20 years. However, there are few quantitative comparisons between model-based approaches and legacy document-centric approaches. Studies have shown that the lack of data regarding improvements of modeling has decelerated the adoption of model-based practices. To help address this gap, the authors conducted a screening experiment to compare the effects of modeling on an engineer’s ability to determine if a system has met its originating requirements. First, a notional acquisition program was created based on an unmanned aerial system, including originating requirements as well as both document and model-based design artifacts. Requirements were captured in both a traditional requirements document and a goal oriented requirements engineering model. System implementation data were capturing in both traditional document artifacts and a SysML model. Participants of varying experience levels used randomized combinations of document and model-based approaches to determine whether the notional system met its requirements. The experiment measured the review duration and accuracy, permitting analysis of the effects of model-based approaches for both requirements and system implementation data. The results of the experiment showed that use of a requirements model did not statistically effect the review. A system implementation model was shown to improve novice participants’ reviews, but did not statistically effect experienced participants. The results of this study should inform future research on the use of models, particularly the return on the modeling investment.