XR Music Videos10th July 2020
Cross-reality (XR) as a platform for media entertainment can be much more than immersive gaming, just as film entertainment is more than blockbuster cinema. Contemporary XR productions offer an expansive array of entertainment value to existing and emergent 3D media practices and beyond. However, this technological intervention can arguably only add further value over more traditional multimedia practices as a supplement to the main creative focus, such as music, cinema, gaming, and so forth, by making full use of the 6DoF experience as well as implementing engaging interactions and introducing player agency. By approaching the use of technology creatively in this way, objects of attention within an immersive virtual environment (IVE) can be presented in such a way that audiences can move around, interact, and engage with creative materials, making the XR experience fundamentally unique and rewarding to the audience. Although highly immersive and engaging, 6DoF also brings to light issues concerning conventional performance practice in and for IVEs and how artists can engage new audiences with the artistic content remaining as intended.
Finding new ways to visualize musical performance is driven by artistic creativity, a desire to technologically innovate, and a need to capture new and existing audience attention. Immersive XR music videos are currently being studied at V-SENSE as an emergent art form in and of themselves as many emergent mixed reality technologies are being applied in this endeavor, such as stereoscopic and 360° audiovisual spatial recording technology. These capture technologies serve to expand the traditional viewing medium to include further dimensions of immersion, interaction, and imagination for the audience and are closely tied to advancements being made in home PC GPU/CPU speeds, HMD optics, software data processing capabilities, and AI.
Volumetric XR Music Videos
Music videos are short films that integrate songs and imagery that are produced for both artistic and promotional purposes. Modern music videos apply a variety of media capture techniques and creative post-production technologies to provide a myriad of stimulating and artistic approaches to audience entertainment and engagement for viewing across multiple devices. Within this domain, volumetric video (VV) capture technologies are becoming a popular means of recording and reproducing musical performances for new audiences to access via traditional 2D screens and emergent XR platforms, such as augmented and virtual reality (AR/VR). These 3D digital reproductions of live musical performances are being captured and enhanced by a variety of video effects that deliver cutting edge audio-visual entertainment. However, the precise effects of VV in music video entertainment has yet to be explored from a user perspective. In this project, we will show how users respond to VV representations of music performance in both AR and VR. Our results will demonstrate how audiences are likely to react to music videos in an XR context and offer some insight into how future music videos may be developed. We anticipate our findings will be a formative starting point for more sophisticated, interactive music videos that can be accessed and presented via emergent XR platforms. For example, how VV XR music videos may be captured, edited, and accessed for live performance reproductions across multiple platforms.
Haptics in Music: Exploring the potential of audio-tactile vibrations in experiences of cross-reality media.
Sound and touch are innately linked and can, therefore, be explored together in human-computer interactions that involve music and emergent cross-reality (XR) technologies. As the auditory manifestation of sound, vibrations allow us to listen to music not only with our ears but with our whole body too. This simple act of acoustics provides multimodal stimulation to both performers (active) and audiences (passive) alike. At a music performance level, a complex system of auditory–haptic grammar has been established between music, musicians, and audiences that can render subtle differences in the experiences of a musical performance. This presentation aims to explore previous research on the topic of digital musical instruments, audio-tactile feedback, and the relationship between audio-induced vibrations on the body and experiences of music. This serves to build knowledge of the psychophysical systems at work during musical interactions that can be implemented to generate both music and vibration for remote experiences of music performance in virtual reality (VR).