Estimating cognitive load from speech gathered in a complex real-life training exercise

Abstract Speech-enabled applications are becoming prevalent, providing opportunities for real-time detection of speaker characteristics. Estimation of cognitive load from speech is one type of speaker characteristic that can provide insight into the human state in complex, highly dynamic human-machine teaming scenarios and be used to adapt interaction with the user to their current cognitive state. Cognitive load estimation from speech experiments are typically performed on speech gathered in laboratory settings. By contrast, this research is performed on a real-life dataset that was not created for the purpose of cognitive load assessment. Speech was extracted from recordings of a military simulation exercise in which air battle managers communicated with pilots flying simulated aircraft. This paper assesses whether cognitive load can be estimated from speech self-labelled by exercise participants and collected in a realistic setting, and examines how well cognitive load estimation methods translate from the laboratory setting to the real-world. Analysis suggests that participants’ self-assessment of workload at periodic intervals can be used to label speech to create 2-class cognitive load classifiers. The analysis also shows that including some target speaker speech in speaker independent training data results in higher classification accuracy than when classifiers are built solely from speaker dependent data.