How 3D Sound Makes Virtual Reality More Real

Attend a virtual reality meetup or conference and the discussion will eventually turn to developing better 3D spatial sound for VR experiences. The New York Times’ VR journalism platform, NYTVR, recently upped the ante (for iOS, Android and Google’s VR platform Daydream) when Tribeca-based Q Department Studio, creators of a VR and augmented reality spatial sound system called Mach1, teamed up with Secret Location—makers of the VR content management system, VUSR—to help bring virtual sound up to speed with visuals.

As Q Department Studio’s Jacqueline Bošnjak tells The Creators Project, she and Mach1 creator Dražen Bošnjak wanted to enter the VR sound arena so that the VR and AR experiences could become more holistically immersive. She says that while sound—as with everyday reality—is half the “presence” for more complete immersion, it lags behind VR’s visuals.

Dražen, an inventor originally from Bosnia known for the immersive quality he brings to his musical compositions and sound design, wanted to bring cinematic quality sound to VR, and developed Mach1 to, ideally, do just that. He developed Mach1 while working on Ridley Scott and Robert Stromberg’s MartianVR experience, as well as the 13-minute Mr. Robot VR experience for Sam Esmail, and other VR projects.

“[H]igh production value is critical in VR and that is only possible if the playback formats support that quality,” says Jacqueline. “Mach1 is the first audio format created specifically for virtual reality and augmented reality… [bringing] cinematic quality sound to VR.”

As Dražen notes, the ability to sense acoustic vibration is one of humanity’s five basic senses—one that helped us survive predators, hunt for food, and locate loved ones. Some aspects of high quality sound require a trained ear to appreciate, while other sounds work through a basic smartphone speaker that is compressed over a wireless signal. He's amazed at the diversity of audio sensations in human hearing, and wants VR and AR audio to reflect this dynamic spectrum.

“Some aspects of visual technologies, mostly at the level of post production and processing are ahead of sound,” says Dražen. “It is much easier to trick the brain that it is hearing the right sound than it seeing the right image therefore exponentially larger amounts of money and time are designated towards developing and processing believable visual effects than sound.”

“Up until now there was more incentive to invest in the visual side, but that is rapidly changing with virtual reality,” he adds. “We are now creating a world where the sound is an equal counterpart to the visual.”

Dražen points out that at the level of recording (or, filming) and reproduction, sound is not at all behind visual technologies. But VR demands that spatial audio recording and reproduction techniques move ahead of visuals. Or at least that is what Dražen is hearing from VR and AR designers.

“There is a growing need for high quality sound,” he says. “Directors of VR/AR are now professing that sound is playing a more important role than ever and that it makes up for as much as 60% of the overall experience.”

With Mach1, Dražen says that VR designers get a variable depth within the 3D sound field. He says it’s different than the variable depths inside a 3D sound field based on a game engine’s spatialized sound. A self-professed huge gamer, Dražen thinks gaming’s spatialized sound can be “jarring,” “dead,” and “unimaginative.”

For Dražen, getting VR and AR sound up to speed requires bringing all knowledge from film sound production and post-production, music recording and production, and game sound production to the table, then adapting it to suite virtual or augmented reality needs. Above all, he believes that VR and AR media require strong control of sounds for the purposes of virtual storytelling.

“We took an approach practiced in music and film production and evolved this into a 3D sound environment,” Dražen explains. “We designed for the transparent sounding system and workflow tools—no artificial coloration, signal processing, or degradation. The mix achieved during the final mix—the creative mix—is the technical mix and would play back exactly the same during VR/AR playback.”

“Nurturing our eye and ear sensors with the qualities worthy of hearing our human voices sing, a subtle breeze in a tree, a water spring, an owl, strums of an acoustic guitar and the distortion of an valve amp,” adds Dražen, “we have the capabilities to do this, and now VR and AR demand quality to reinforce the suspension of disbelief.”

The VR and AR gauntlet has been thrown down. With no potential shortage around the globe of tech innovation in the realm of sound, it should be interesting how any VR sound wars play out.

Click here to experience VR projects that feature Mach1’s spatial audio playback format.

Related:

This Performer Turns Musical Dreams into 3D Worlds

The Legendary Animator Behind Star Wars and Jurassic Park Tackles VR

I Fought Aliens (and Died) at a Multimedia Festival