DARCAAT: DARPA Competence Assessment and Alarms for Teams

Assessing teams in complex military environments requires effective tracking of individual and team performance. Indeed, performance measures must be both accurate and timely in order to provide effective real-time alarms. However, current methods of monitoring team and group performance often rely on delayed outcomes or global metrics that are insufficiently detailed to detect failures until recovery is impossible, and are often unable to reveal the causes of failures. An untapped source of timely and diagnostic information lies in the communications among team members. The DARCAAT program developed and tested a toolset for automating team assessment and near real-time alarms. The toolset uses Automated Speech Recognition and Statistical Natural Language-based techniques for embedding automatic, continuous, and cumulative analysis of team communication in training and operational environments. The techniques include measures of the content, patterns, and style of team members’ communications. These measures were combined using machine learning techniques to develop performance models based on Subject Matter Expert (SME) ratings of teams. Focusing on the domain of convoy training, we collected team performance and communication data from the Fort Lewis DARWARS Ambush! convoy training virtual environment and from the National Training Center’s live convoy STX lane training. Tests of the performance models and critical incident detection capabilities showed that the technology agreed significantly with SMEs’ ratings of teams, and could identify a majority of the team critical incidents. In this paper we discuss the implications for modeling team performance based on communication, describe the development of the technology, and demonstrate how it can process communication to detect critical incidents and to generate team performance metrics. Finally we describe how this technology can be integrated into training systems for automatic team assessment. These systems can provide automated feedback and can alert teams and commanders of potential problems before they occur.