A performance by band LIVYATANIM gets transformed into a binaural spatial audio experience.
GIF courtesy of Or Fleisher and LIVYATANIM
Virtual reality and music make for a fruitful combination, and a new spatial audio piece by Or Fleisher and Aviv Meshulam sees Meshulam's band LIVYATANIM performing live using binaural sound to add an extra level of immersion to the VR experience. The pair have previously collaborated on Myth, an audiovisual VR film that let users explore music as topographical world populated by glitching whales and birds.
For this new venture, SONO, Fleisher and Meshulam wanted to captured a live performance in a unique way one that not only immersed the viewer but also featured a number of audio-reactive visuals. As LIVYATANIM perform, they appear to be floating in space on a small rock ("We chose the moon as our venue," notes Meshulam) while celestial happenings, reacting to the music's rhythm, take place around them.
"We decided to use lights that react to each musician’s musical role," notes Fleisher. "Emphasizing it and visualising the spatial audio element in the experience. We used the musical parts to drive the environment in the form of non-realistic massive celestial events."
The piece can be experienced online in a browser with or without a VR headset, although it's recommended that you wear one if you've got it. To create the virtual moon landscape setting, the creators called upon NASA’s open source library of 3D scans, focusing in particular on the Copernicus Crater (above). The musicians' performances were shot separately using a cinematic stereo shooting rig and processed into their 3D environments, while the audio-reactive elements were built in TouchDesigner.
"After exploring how music and VR can merge into a cohesive and natural piece in Myth, binaural sound was an obvious step forward for making VR experiences truly realistic and immersive," explains Fleisher. "So naturally we wanted it to be an integral part of SONO. The project began as a desire to capture a live performance of the band in a new way, one that puts the audience in the center of things both visually and sonically."
Learn more in the making-of video below.