Immersive Orchestras: audio processing for orchestral music VR content

TitleImmersive Orchestras: audio processing for orchestral music VR content
Publication TypeConference Paper
Year of Publication2016
Conference NameVSGAMES 2016 - 8th International Conference on Virtual Worlds and Games for Serious Applications
AuthorsJaner, J., Gómez E., Martorell A., Miron M., & de Wit B.
Conference Start Date07/09/2016
Conference LocationBarcelona
Keywordsorchestra, source separation, virtual reality
AbstractThis paper combines Audio Signal Processing and Virtual Reality (VR) content to create novel immersive experi- ences for orchestral music audiences. In VR, the auralization of sound sources of recorded live content remains still a rather unexplored topic. We aim to build a multimodal experience, where visual and audio cues bring a sonic augmentation of the real scene. In the particular scenario of orchestral music content, our goal is to acoustically zoom on a particular instrument when the VR user stares at it. This work aims to improve the learning aspects of music listening, either for education or for personal enrichment. We use audio signal processing to separate different sound sources (instruments) in a acoustic scene (orchestral music recording). Given the signals captured by multiple microphones and the musical score of the piece, our system is able to isolate the different instruments. From the processed separated tracks, we use a binaural rendering technique to emphasize a give instrument. For these experiments we used original content from top European orchestras.
preprint/postprint document