Feel What You Don't Hear: A New Framework for Non-aural Music Experiences

Just as the way a performer is moved differs even among audiences who have the same impression of the performance, the sensations and experiences felt by the performers themselves and the audiences' experiences also differ. The purpose of this research is to create a new listening experience by analyzing and extracting the performer's introspection of rests, groove, and rhythm, and physically presenting it to the audience. Although these elements are important in shaping music, they are not always directly expressed as auditory sounds. Our hypothesis is that this introspection, such as a sense of rhythm and groove, is latent and observable in physiological states such as breathing and heartbeat. By sensing and presenting them to the audience, music appreciation that includes introspection could become possible. In other words, by sensing and presenting introspection to the audience, the music listening experience itself can be redesigned to include a physicality that is closer to the performer's experience of the music, rather than being passive in an auditory sense. In this study, preliminary experiments were conducted on the extraction of the performer's introspection, and a device was designed to present it to the audience.