The entire premise of Virtual Reality (VR) relies on the user buying into the illusion and immersing themselves into whatever world has been created for them. But if there is a disconnect between the visual elements and the audio – it cannot really count as a fully immersive experience. Still, not much attention gets given to the big role audio plays in engaging the user – with much of the focus tending to be on what visual experiences can provide instead.
“Sound is lacking behind the visual. The focus of the industry has always been on the visual,” Vincent de Koning, Creative Director at Big Orange says when he talks to me over the phone. “Audio triggers fantasies and visual things don't trigger fantasies because it’s filled in already. So to trigger emotion, it's easier done by audio then by picture.”
But things are slowly starting to change. Earlier this year, Facebook acquired Two Big Ears, an Edinburgh-based immersive audio company that focuses on how sound plays in 3D spaces. Google is experimenting with a technology called Omnitone that uses spatial audio to replicate “how sound waves interact with the environment and the listener’s head and ears”. Google says spatial audio is “essential for providing a valuable VR experience”.
Why is VR audio so complex?
Dr. Gavin Kearney, Lecturer in Audio and Music technology at the University of York is of the opinion that video tends to get prioritised over audio: “The computational real-estate given to the audio is usually very small and it is difficult to do nice immersive audio renders with such limited computational power,” he says.
Another added complexity is how our ears process sounds in unique ways.
“Your ear is as unique as your fingerprint. If I put some microphones in my ears and I recorded sounds coming from in front of me when I play it back over headphones to you, you may hear the sound coming from behind you,” Kearney explains.
Kearney says the use of head-tracking can overcome a lot of the problems in personalising audio experiences.
“By head tracking when you move your head, the phone knows about head rotation and tilt and so forth. So it then can change the properties of the sound fields or what's reproduced to the headphones to give a stable sound field. So you think that sounds are not only outside of your head but are at a stable position in 3D space.”
Google has taken note of web-based VR projects like WebVR, which brings VR experiences to the web and so has already started experimenting with the role of spatial audio in adding the immersive dimension to the experience.
Kearney, along with his former colleagues, has worked with Google on the Omnitone project and describes how it works.
“Ambisonics tricks the brain into thinking that if the head moves one way and the sound field is rotated the other way, it thinks that the sounds are staying in the same position in space. So it's updating in real-time and it's so easy to rotate the sound field in a counter motion to the way the head is moving and that gives you the impression of a stable sound source. This is why ambisonics is finding its way into VR frameworks so nicely. It’s why Google is using it and it's going to play a big part in how sound is going to be reproduced for the foreseeable future,” Kearney says.
What about the headphones themselves?
Big, clunky headphones can also ruin the full VR immersion experience. Kearney explains he is currently assessing whether headphones take you out of the VR experiences versus listening to sound through a loudspeaker.
“Imagine you are wearing your Oculus Rift or HTC Vive and you can be listening over a real loudspeaker rig that's playing this real acoustic environment or you can listen through headphones. Will headphones affect your suspension of disbelief? Will it take away from the convincing aspects of the rendering or does it matter?” Kearney poses.
Custom earphones have been around for a long time and could be a solution. Not only are they tailor-made to our ears, they are also lightweight. But it can be costly to get ear moulds done from an audiologist.
Mark O'Callaghan, founder of Wavz earphones, tested a number of high-end earphones on the market and found himself disappointed with the audio experiences provided by each of them. One day he saw a poster in an audiologist office explaining the way audio waves work.
“I went to the guys that made my earphones and told them that I really wasn’t impressed with them and asked if I could do something about this. So they were like, ‘That’s cool’, gave me some supplies and I played around with it,” O'Callaghan explains when I catch up with him at Pauls Café in London.
O'Callaghan offered to create for me my own custom "Dredds" earphones with a “special ingredient in the silicon” so I could test them out. He filled up my ear canals with silicon using ‘foam blockers’ to prevent the silicon from going all the way down the eardrum. I was then told to let the silicon set for five-ten minutes.
We met up again a week later at Scarfes Bar in London to demo them. At the moment the earphones are just hand-made prototypes using Eo320 drivers and O'Callaghan is looking for investment with the intention to eventually use 3D printers to create more accurate, higher volumes and personalised earphones for customers.
The earphones fit my ears comfortably and although they did not block out sounds around the room completely, the audio did feel quite immersive. I tried out Audio Illusion from YouTube which mimics binaural audio where the sounds go around your head at different moments. For the VR experience I tried Battle 360 where I battled various airplanes in the sky. The earphones also tested favourably with music against the Final Sonorous VI headphones priced at £549 ($725) in Selfridges, London.
“All of my secret is in how the silicon is done. It’s got nothing to do with the speakers which is why I am trying to raise some funds to develop the speakers, get the 3D printers and all that kind of stuff,” O'Callaghan tells me.
What is the future of audio VR?
Kearney would love to see a world where we can all get our own personalised 3D binaural audio experiences.
“You could go into the backroom of Specsavers with an acoustic booth with a whole load of loudspeakers set up in the room. You would just put little microphones in your ears and the loudspeakers will put out these little chirps and measure how the sound impacts around your head. From that you will leave with a USB key which would have your binaural filters and then you could upload them to your phone or PC and experience your own pre-set or three dimensional binaural sound on whatever application you want,” Kearney explains. “There is already an international framework for storing such data called SOFA format so it's not unreasonable to think that in the future that this could be something that could happen.”
But Kearney notes that the “golden egg” of all this would be to eventually find a set of filters that work for large numbers of people, which he admits is a tricky prospect: “I still think personalisation is a good way to go - at the very least users should have the choice of the best binaural filters for them in the VR application.”
Koning sees audio playing a really useful role as a form of treatment for treating phobias such as claustrophobia: “If you hear you are in a tight room, the audio can only ignite the claustrophobia. The form of therapy could make the sounds more claustrophobic to see if there can be a solution or training.”
Kearney concludes: “A lot of people talk about audio in VR as representing reality as closely as possible. One thing that cinema and television has told us – is that actually reality is boring. If you watch the latest Bourne movie and the sound effects reflected real life, it wouldn't be very impactful. From a sound design perspective, there should be an awareness of virtual hyper reality not Virtual Reality because I think that's an important aspect that I don't hear enough people talking about in terms of sound reproduction.”
PREVIOUS ARTICLE«World watches Nutanix and waits for Big Chill on IPOs to end
NEXT ARTICLETelecom APIs could increase African innovation»
Phil Muncaster reports on China and beyond