“Pool of Toys”: Multimodal Visual Music Rendering of Sequenced and Real-time Control Events

The goal of this study is to display music visually to aid recognizing each component, enjoying music not just aurally but visually as well. Elements of western music comprise multiple aspects (e.g., rhythm, melody, harmony, and orchestration). To express them, we introduce a “live song” called “Pool of Toys.” A pool filled with toys is used as a motif in a game engine (Unity). A multitimbral song including musical elements was created using DAWs (GarageBand and then Ableton Live). Each part is imported as a MIDI track into Unity, controlling the attributes of visual elements (floating toys) at runtime. A surface grid controller (Launchpad Pro) is used at runtime along with the precomposed music so that performers can synchronously play the MIDI terminal with output displayed in realtime.