Applying Model-Based Validation to Inference Enterprise System Architecture Selection

In this paper, we describe a framework for comparing and selecting inference enterprise models. An inference enterprise is an organizational entity that uses data, tools, people, and processes to make mission-focused inferences. Intuitively, organizations could organize around one of several inference enterprise models to make the same inference. To address the inference enterprise model selection problem, we combine multi-inference enterprise modeling, model-based validation, and statistical inference to rank order inference enterprise candidates. Inference enterprise multi-modeling affords us the opportunity to simulate representative data set to the organization’s mission. Model-based validation employs normative decision theory to score empirical results using a utility function, and statistical inference allows us to generalize candidate rank ordering. We demonstrate the framework described in this paper and compare expected utility-based rank ordering with rank ordering based on expected F1 score. Using generic performance metrics such as F1 potentially has adverse impacts to an organization’s mission.