Object and Gesture Recognition to Assist Children with Autism during the Discrimination Training

Teachers prompt children with autism to redirect their attention to the object discrimination training and reduce the time they spend “off task”. In this paper, we describe MOBIS, a mobile augmented reality application enabling multi-modal interaction to provide guidance to students with autism during the object discrimination training. The system uses a vision-based object recognition algorithm to associate visual and verbal prompts to the object being discriminated (i.e., “object of interest”). The results of a performance evaluation of the system show that the object recognition component achieves an accuracy of 90%, processing an image every 0.5 seconds. Accelerometers placed on objects of interest are used to detect interaction gestures with an accuracy of 87%. The performance of both algorithms is sufficient to support the object discrimination training in real-time.