How can we assess whether our simulation models improve the system understanding for the ones interacting with them

Most of the system dynamics studies that evaluate decision making in complex dynamic task focus on the evaluation of performance over repeated trials and on the effectiveness of different instructional strategies as far as performance is concerned. Especially when a strategy seems to yield promising results in terms of performance, it becomes essential to know whether improved performance is due to improved system understanding, i.e. to correct rules or due to other reasons such as trial and error. This paper contributes to the emerging literature in system dynamics about assessing system understanding. Based on the way experts make decisions we develop a step by step guide to evaluate how the understanding of the system develops in the course of subjects interacting with the system through a simulation model. We apply our guide to the reindeer management task and analyze data from previous experiments with the task. This application provides important insights for the further development of the questionnaires that are applied for assessing understanding.