Development of a Socially Assistive Robot Controlled by Emotions Based on Heartbeats and Facial Temperature of Children with Autistic Spectrum Disorder

A socially assistive robot (termed N-MARIA – New Mobile Autonomous Robot for Interaction with Autistic) is being developed in UFES/Brazil to help the Autism Spectrum Disorder (ASD) therapy. This robot has speakers and color (RGB) and infrared cameras, allowing it to emit artificial voice (to motivate the children) and capture RGB and thermal images from the children to infer their emotions. The robot can operate autonomously (using its onboard sensors and a behavior-based control system) or be remotely commanded by a therapist. The controller uses data from a laser sensor to move towards the children until a social distance (according to the concepts of proxemics), with a mean error of 1.5% of the distance. Images from the children are used to infer their emotions, and thus adjust the robot behavior. The thermal camera gets information about their facial temperature (through the technique of Infrared Thermal Imaging – IRTI), and the RGB camera gets information about their Heart Rate Variation – HRV (through the technique of remote photoplethysmography – rPPG). The rPPG technique has sensitivity of 3 bpm when the child is static, and 15 bpm when the child is moving. Using HRV, it was possible to discriminate arousal (passive to active emotion) and valence (positive to negative emotion) with mean accuracy of 34% and 45%, respectively. On the other hand, the IRTI technique was used to determine the facial temperature (sensitivity < 0.07 °C) of the children, with a mean accuracy of 85.75% to determine five emotions. Both emotion estimation (based on rPPG and IRTI) are then used as input for a Decentralized Kalman Filter (DKF) and an Information Filter (IF) to infer the children’s emotion, and change the robot’s behaviors using an architecture operating dynamically for behavior selection.