Measuring Situation Awareness with Probe Questions: Reasons for Not Answering the Probes

Abstract Situation Awareness (SA) refers to an individual's understanding of what is happening in their environment, and what is likely to happen in the near future [1]. SA is an important construct to study in aviation because lower levels of SA are associated with increased aviation accidents and mishaps [2]. One technique to measure operator SA is through the use of online probe questions, where operators are queried about their task environment at various intervals while performing the task. The current study used the Situation Present Assessment Method (SPAM) [3] for administering online probes. With SPAM, a ready prompt appears on a panel and participants are instructed to accept the prompt when workload permits to be presented with a SA probe question. Latencies in accepting the ready prompt are indicative of the operator's workload level at that time. When ready prompts go unanswered, (ready time-outs; RTO), it is assumed workload is too high. When a prompt is accepted, probe questions are presented for a limited time (60 seconds in this study). Thus it is possible they too will go unanswered by timing out (question time-out; QTO). QTOs can occur for the following reasons: 1) operator did not know the answer, 2) due to the dynamic changes in the task, the operator workload became too high to devote resources to answering the question, 3) question was unanswerable due to the current context, or 4) question required more time to answer than allowed. In the present study, we analyzed a subset of the data from a larger study (see Brandt et al. [4]) to examine reasons for why probe questions go unanswered. For QTOs, the majority of questions went unanswered because the operator did not know how to answer the question, or because the operator was dealing with high workload. RTOs, which occurred more frequently, were normally due to task/workload.