Continuous auditory feedback for sensorimotor learning

Our sensorimotor system has developed a specific relationship between our actions and their sonic outcomes, which it interprets as auditory feedback. The development of motion sensing and audio technologies allows emphasizing this relationship through interactive sonification of movement. We propose several experimental frameworks (visual, non-visual, tangible, virtual) to assess the contribution of sonification to sensorimotor control and learning in interactive systems. First, we show that the auditory system integrates dynamic auditory cues for online motor control, either from head or hand movements. Auditory representations of space and of the scene can be built from audio features and transformed into motor commands. The framework of a virtual sonic object illustrates that auditory-motor representations can shape exploratory movement features and allow for sensory substitution. Second, we measure that continuous auditory feedback in a tracking task helps significantly the performance. Both error and task sonification can help performance but have different effects on learning. We also observe that sonification of user’s movement can increase the energy of produced motion and prevent feedback dependency. Finally, we present the concept of sound-oriented task, where the target is expressed as acoustic features to match. We show that motor adaptation can be driven by interactive audio cues only. In this work, we highlight important guidelines for sonification design in auditory-motor coupling research, as well as applications through original setups we developed, like perceptual and physical training, and playful gesture-sound interactive scenarios for rehabilitation.