Automatic assessment of cognitive and emotional states in virtual reality‐based flexibility training for four adolescents with autism

Tracking students? learning states to provide tailored learner support is a critical element of an adaptive learning system. This study explores how an automatic assessment is capable of tracking learners? cognitive and emotional states during virtual reality (VR)-based representational-flexibility training. This VR-based training program aims to promote the flexibility of adolescents with autism spectrum disorder (ASD) in interpreting, selecting and creating multimodal representations during STEM-related design problem solving. For the automatic assessment, we used both natural language processing (NLP) and machine-learning techniques to develop a multi-label classification model. We then trained the model with the data from a total of audio- and video-recorded 66 training sessions of four adolescents with ASD. To validate the model, we implemented both k-fold cross-validations and the manual evaluations by expert reviewers. The study finding suggests the feasibility of implementing the NLP and machine-learning driven automatic assessment to track and assess the cognitive and emotional states of individuals with ASD during VR-based flexibility training. The study finding also denotes the importance and viability of providing adaptive supports to maintain learners? cognitive and affective engagement in a highly interactive digital learning environment.