Hierarchical Goal Analysis of Dynamic Decision Making in Microworld Experiments

Abstract : Recent developments in microworld-based experiments provide researchers with an opportunity to conduct complex and dynamic experiments in laboratory-controlled environments, thus narrowing the gap between laboratory-based and field experiments. The performance assessment in a dynamic decision making environment, however, requires new methods for evaluation and analysis of data and cognitive systems. This memorandum discusses the application of Hierarchical Goal Analysis (HGA) to evaluate cognitive systems in a distributed team environment. The process of conducting HGA involves the following steps: a) derivation of goal hierarchy, b) assignment of goals to subjects, c) identification of controlled variables, and d) completion of templates that specify goal attributes. The HGA-derived controlled variables provide additional measurements of performance that are closely related to subjects' decisions. We conducted upward information flow and stability analyses to evaluate the system that the subjects were functioning in. The analyses helped to identify a number of situations that might impede subjects' performance during task execution. Finally, this memorandum discusses the potential benefits of applying HGA in the context of distributed and dynamic simulations and proposes future work to use the HGA outputs as the basis for the development of a computational model for predicting subject performance under specific task conditions.