Audio in VR: Creating 3D Sound

Audio in a virtual reality environment is one of the most important pieces to make a player feel immersed in the experience. As a game is created and designed, sound not only needs to fit the look and feel of the game, it takes on another level of difficulty and expertise when making it 3D for VR. We sat down with Bethany Borden, our resident sound composer & designer, and she shared three things she has learned while creating 3D audio that all sound designers should keep in mind when creating audio for a VR environment. Read Bethany’s tips and experiences in her own words below.

Realism
– the quality of representing a person, thing, or situation accurately or in a way that is true to life

I had an “a-ha moment” at GameSoundCon in LA this year. Chris and I were sitting at a table in one of the rooms where water, coffee, and tea were available, and occasionally a waiter would be walking around carrying a tray.

All of the sudden I heard a loud sound behind me, which alarmed me and caused me to jump. I turned and saw that one of the waiters had dropped his tray, and that eased my mind once I knew what the sound was and that everything was okay.  I thought, “That is why sound is so important in VR! One unknown sound can change your whole mood and almost force you to look in certain directions.” In a virtual reality game, you are there in the room or environment of the game. You can turn 360 degrees in order to see everything, but that leaves a lot that you can’t see when you’re not looking at it. Music cues and well placed sound effects will help users know where to go, what to do, and what or who is coming to help make the experience completely realistic.

When playing PC, console, arcade, and mobile video games, the music and sound effects come out of the speakers or headphones and the most that can happen logically is for you to hear the sound to the left or right, because you can only see left to right. There is nothing above you, behind you, or next to you while you’re playing these games. In virtual reality, the sounds are attached to the object in the game, so if you look at it, it will sound like it’s in front of you. But if you turn to your left, you will hear the sound in your right ear. If you turn 180 degrees, it will sound like it’s behind you. This is a very new concept, so programmers are working hard to make tools that audio teams can use to implement sound, but right now it feels like every man for themselves as we all try to figure out how to create the best experience in VR by making the best SOUND experience in VR.

Keywords:

  • Spatialization – the direction of incoming sound
  • Synthesis – the creation of source sounds

Immersion
– the state of being deeply engaged or involved

One of my favorite games is ‘Serious Sam: The First Encounter! The game has everything I’m looking for: combat, weapons, unique characters, a story, humor, loot drops, hidden items, puzzles, and last but not least, great sound design. At the time I played the game, I was still teaching music and hadn’t written a note for or implemented a sound in video games. I did notice, however, that when I was in a particularly intense combat scene, the transition of music into the dangerous fight would make my whole body tense up and prepare and also make my senses very alert. Once I had killed the last enemy in the wave, the music would gradually dissipate into something calm that signified the end of danger and a sense of relief. It was usually at that time that I even noticed the music that had made me feel the way I felt during the fight. Basically, the idea of immersive sound design is for the player to feel what the music or sound effects intend them to feel before they actually notice them or think about them.

Usually, the music in games is the most effective way to help people connect with the game emotionally.  In virtual reality, sound effects are actually much more important, because they alert you to feel something when you hear them and try to figure out what the sound is and where it’s coming from. Michael Abrash said during his keynote speech at Oculus Connect 2 that the goal in VR is to drive the human perceptual system, which includes manipulating as many of the human senses as we can, resulting in creating the most immersive experience. Sound is one of the areas where there is a pretty clear path as to what needs to happen to get the desired results, but there is not an easy way yet. Audio engineers and programmers still have a long way to go, but leaps are being made every day as more people are researching and experimenting with sound in VR.  

Keyword

  • HTFR (Head-Related Transfer Function)a response that characterizes how an ear receives a sound from a point in space; a pair of HRTFs for two ears can be used to synthesize a binaural sound that seems to come from a particular point in space.

 

Watch this example of HTFR with headphones

Balance
– mental and emotional steadiness

You get motion sickness when one part of your balance-sensing system (your inner ear, eyes, and sensory nerves) senses that your body is moving, but the other parts don’t. This conflict between the senses causes motion sickness. Your eyes see motion, but your body doesn’t sense it. Luckily, many steps have been taken with the visuals and mechanics in Virtual Reality to now prevent most people from feeling any negative side effects like motion sickness while experiencing VR.

The definition of sound is vibrations that travel through the air or another medium and can be heard when they reach a person’s or animal’s ear. Physical vibrations plus sound make you feel like what you’re seeing is real, creating a balanced experience.  When we were creating the bow and arrow for our game ‘The Ancient Remains, I worked very closely with the team to make sure the correct sounds were being implemented, and our lead developer added vibration in the controller while you are pulling the arrow back in order to create tension along with the sound effect. The vibration stops on the release of the arrow while the release sound plays, creating the feeling of the weight of the arrow. But if those sounds and vibrations didn’t match the visuals, the experience would feel slightly off-put, even if you didn’t quite understand why. Sound is 50% of the VR experience, so balance is key!

Keywords

  • Propagation – how sound moves around the space
  • Binaural – recorded through two separate microphones and transmitted through two separate channels so that the reproduced sound seems to surround the listener and to come from more than one source.

Helpful Articles About 3D Audio

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *