XR Music Videos
10th July 2020
Extended-reality (XR) as a platform for media entertainment can be much more than immersive gaming, just as film entertainment is more than blockbuster cinema. Contemporary XR productions offer an expansive array of entertainment value to existing and emergent 3D media practices and beyond. However, this technological intervention can arguably only add further value over more traditional multimedia practices as a supplement to the main creative focus, such as music, cinema, gaming, and so forth, by making full use of the 6DoF experience as well as implementing engaging interactions and introducing player agency. By approaching the use of technology creatively in this way, objects of attention within an immersive virtual environment (IVE) can be presented in such a way that audiences can move around, interact, and engage with creative materials, making the XR experience fundamentally unique and rewarding to the audience. Although highly immersive and engaging, 6DoF also brings to light issues concerning conventional performance practice in and for IVEs and how artists can engage new audiences with the artistic content remaining as intended.
Finding new ways to visualize musical performance is driven by artistic creativity, a desire to technologically innovate, and a need to capture new and existing audience attention. Immersive XR music videos are currently being studied at V-SENSE as an emergent art form in and of themselves as many emergent mixed reality technologies are being applied in this endeavor, such as stereoscopic and 360° audiovisual spatial recording technology. These capture technologies serve to expand the traditional viewing medium to include further dimensions of immersion, interaction, and imagination for the audience and are closely tied to advancements being made in home PC GPU/CPU speeds, HMD optics, software data processing capabilities, and AI.
Volumetric XR Music Videos
Music videos are short films that integrate songs and imagery that are produced for both artistic and promotional purposes. Modern music videos apply a variety of media capture techniques and creative post-production technologies to provide a myriad of stimulating and artistic approaches to audience entertainment and engagement for viewing across multiple devices. Within this domain, volumetric video (VV) capture technologies are becoming a popular means of recording and reproducing musical performances for new audiences to access via traditional 2D screens and emergent XR platforms, such as augmented and virtual reality (AR/VR). These 3D digital reproductions of live musical performances are being captured and enhanced by a variety of video effects that deliver cutting edge audio-visual entertainment.
Audience Experiences of a Volumetric Virtual Reality Music Video.
The precise impact of volumetric video in virtual reality music video entertainment has yet to be fully explored from a user’s perspective. In our IEEE VR 2022 paper we show how users responded to volumetric representations of music performance in virtual reality using the Björk “Notget VR” experience from her Vulnicura Virtual Reality Album. Our results preliminary demonstrate how audiences are likely to respond to music videos and offer insight into how future music videos may be developed for different user types. We anticipate our essay as a formative starting point for more sophisticated, interactive music videos that can be accessed and presented via XR technologies. For example, how VV XR music videos may be captured, edited, and accessed for live performance reproductions across multiple platforms.
Creating the V-SENSE Volumetric Music Video Experience
Download the Virtual Reality Volumetric Music Video: Featuring New Pagans [VR Demo] here:
https://drive.google.com/drive/folders/1oEK3lFFvH453mkc54WIccJsLPdHV_xrE?usp=sharing
Contact YoungGa(at)TCD.ie for further information.
Haptics in Music: Exploring the potential of audio-tactile vibrations in experiences of cross-reality media.
Sound and touch are innately linked and can, therefore, be explored together in human-computer interactions that involve music and emergent cross-reality (XR) technologies. As the auditory manifestation of sound, vibrations allow us to listen to music not only with our ears but with our whole body too. This simple act of acoustics provides multimodal stimulation to both performers (active) and audiences (passive) alike. At a music performance level, a complex system of auditory–haptic grammar has been established between music, musicians, and audiences that can render subtle differences in the experiences of a musical performance. This presentation aims to explore previous research on the topic of digital musical instruments, audio-tactile feedback, and the relationship between audio-induced vibrations on the body and experiences of music. This serves to build knowledge of the psychophysical systems at work during musical interactions that can be implemented to generate both music and vibration for remote experiences of music performance in virtual reality (VR).
Outputs
Young, G. W., O’Dwyer, N., Vyas, B., and Eagan, D. (2022). Live Virtual Performances in Real-Time Game Engines [MOCAP & VR Demo]. Human Lab stage of Mindfield at Electric Picnic, Ireland.
Young, G. W., O’Dwyer, N., and Smolic, A. (2022). A Virtual Reality Volumetric Music Video: Featuring New Pagans [VR Demo]. At the New Interfaces for Musical Expression (NIME) conference, Auckland.
Young, G. W., O’Dwyer, N., and Smolic, A. (2022). A Virtual Reality Volumetric Music Video: Featuring New Pagans [VR Demo]. At the ACM Multimedia Systems (MMSys) conference, Athlone, pp. 669 – 670.
Young, G. W., O’Dwyer, N., and Smolic, A. (2022). A Virtual Reality Volumetric Music Video: Featuring New Pagans [VR Demo]. At the Sound, Music, and Computing (SMC) conference, Saint-Étienne.
Young, G. W., O’Dwyer, N., Moynihan, M., and Smolic, A. (2022). Audience Experiences of a Volumetric Virtual Reality Music Video. At the IEEE Virtual Reality and 3D User Interfaces (IEEE VR) conference, Christchurch, pp. 775 – 781.