An Interactive Musical Scale Presentation System via Tactile Sense Through Haptic Actuator

This paper proposes an interactive musical scale presentation system that uses tactile sense via a haptic actuator for vocal music training of deaf-blind or hearing-impaired individuals. In the proposed system, we applied the Nintendo Switch as an interactive interface to present the musical scale of songs to the learner. The partner (player/teacher) plays music on a keypad panel, such as a piano keyboard set, on the touch screen of the Switch. The learner receives vibration patterns corresponding to elements of the musical scale including pitch, timing, and melody with the standard game controller of the Nintendo Switch, Joy-Con, equipped with a haptic reactor, and experiences the music performance. Furthermore, in addition to the interactive performance mode, an automatic performance mode was implemented to enable self-learning by the learners.