Oculus development idea with Binaural Audio

Yeah, been hearing a lot of rants in Creative's name in forums such as these. Game Audio really needs a push, especially with Oculus opening up the scenario to multiple possibilities of immersive experiences.

The problem is indeed not simple to solve.

You may find with recordings like the barbershop, matchstick, etc., that it is much easier to fool the listener around behind their head than in front of it. This is largely due to psychoacoustics, in other words the ear components make up the PRTF, while the measurements of head to shoulders, unique shapes of individual's ears and other factors make up a HRTF.

Typically the only way to capture these authentically is in an anechoic chamber with a hemispherical array of speakers and microphones in the ears of the subject.

There is research in more consumer-friendly methods of obtaining personalized 3D audio filters but it is very secretive as the results would be extremely valuable.

Sony claims to have something but since they likely don't have good enough hardware (on the market anyway), they are probably working on something about as gimmicky as Tiamat 7.1, the binaural stereo headphone software products (Creative, etc.).

The hardware to solve this problem is just coming onto the market now, but the team of people solving it will have to be quite good at 3D physics, math, animation, and theoretical 3D audio (a tough combination to get on a team).

The best place to do this kind of mixing is in the game engine or audio API. Turning 5.1 or 7.1 into binaural (as all hardware and driver solutions do) reproduces the feeling of sitting in a room with a 5.1/7.1 speaker system at best.

If your audio mixer can take the actual 3D spatial parameters of the sound and listener and use those to drive a HRTF outputting the final stereo then that is where you can get a sense of the actual XYZ position that a sound is coming from.

The problem is, we all have different ears. Taking a 3D scan of someone's ear shape and turning that into HRTF coefficients is a hard problem, and most people don't have the hardware to gather the necessary data. As a result, its something that works like magic for some people, and not at all for others.

Incidentally, I had a Rob Zombie CD when I was a kid that had one particular repeating sound sample of a yell that my brain totally believed was coming from downstairs. First few times I took off my headphones and went to go look before I realized it was a weird coincidence in the acoustics of the recording environment. When this stuff works its freaky.

Anyone tried this one out before?

http://www.twobigears.com/index.html

Edit: I've downloaded the alpha and so far it looks very promising. They go in to BETA on the April 14th so you can add more than 2 audio sources. Subscription pricing is a little bummer but it's also very cheap and it's still free for non commercial use.

I'm going to run some performance tests tonight and see what the impact is and how much it's worth using.

The individual shape of your ear does have some iimpacct on your ability to locate sound, but allot of it just comes from the delay between sound hitting one ear and the other. All binaural recordings I have listened too have sounded pretty amazing to me so whatever generic ear model they are using is good enough for me. Another factor is sound reflections off wall and other surfaces simulating that may also help make the sounds sound more real. Surely the Aurial patents Creative bought up have expired by now.

Delay doesn't give you binaural, that's just stereo. And it only gives you one dimension of spatial information, not much more than what panning gives you.

Reverb helps but is not reliable. Your brain has to learn how sound reflects and diffracts around a given environment before it can accurately place the echoes. So you can probably precisely place sounds in and around your home, because you have heard lots of sounds coming from known locations, but in an unfamiliar environment your brain can only guess. Computing accurate reverb is also very very expensive. Its basically ray tracing.

Locating sounds is why our ears have a shape at all. Otherwise they would just be holes in our heads, like with non-mammals. You are evidently one of the lucky ones with an average ear shape, but there is a pretty wide range even within a single racial group.

"owenwp" wrote:
Delay doesn't give you binaural, that's just stereo. And it only gives you one dimension of spatial information, not much more than what panning gives you.

Nope stereo just adjusts the volume on the two channels, delay only works with headphones and is one of the key parts of binaural audio. The delay not only lets tell which direction its coming from but lets you tell how far away it is by triangulation.

Perhaps audiophile shops could have a scanner that scans the shape of your ear and Emails the file to you. There is also some nifty software out there that can construct 3d models from multiple camera angles.

"mrmonkeybat" wrote:

Perhaps audiophile shops could have a scanner that scans the shape of your ear and Emails the file to you. There is also some nifty software out there that can construct 3d models from multiple camera angles.

I suspect we could get pretty close to a "personalized" transfer function by merely going through a simple audition process. If the user was supplied with a number of different choices, and a way to easily demo and compare them, they could decide for themselves which is the best match for their physiology.

In some other binaural thread in this forum, someone posted a website that contained many (more than 50) HRTFs, and handily, a sample of rendered audio for each of those TFs. The sample was just a recording of sound "travelling" in a circle around the head. It was easy to tell after listening that not all HRTFs are created equally. Clearly, some sound much more "accurate" than others.

I speculate that a good method for personalizing a HRTF/PRTF would be to give a person ~10 examples of a sound traveling in a circle at ear level. Have the person choose the example that sounds the most correct. Then give the person another ~10 samples to audition, this time with the sound traveling in a circle that goes above and below the person's head. Again, the person chooses the one that sounds the most convincing. Then, some magic would happen that would create or choose a transfer function that hopefully incorporates the best qualities of the two samples that the person chose. (I have no idea how that last bit would be done, exactly.)

EDIT: Here's the .wav files in question if you want to download a few and listen to the variation. (They really need to package all of them up into a single .zip!)

A while back we had some threads on this. Binaural audio can be added now and I don't think we have to worry about individualized HRTFs (see this post: https://developer.oculusvr.com/forums/viewtopic.php?f=26&t=617&start=80#p18714). I'm looking into synthesizing a universal virtual ear (which would give supernormal hearing acuity), but right now we can settle on the standard KEMAR set, such as used by default with OpenAL-Soft. The important thing is that we adapt. Ideally we shouldn't be switching HRTFs in virtual worlds though. I expect our brains have enough cues to distinguish between reality and VR and maintain separate audio-cuing. Regularly switching HRTFs in VR I would expect would lead to bad training -- and poor VR hearing -- as we confuse the two (or more) sets of cues within the same overall "environment" (VR).

I'm seeing more mention of canned binaural... here, reddit, kickstarters... I mean pre-recorded, like the barbershop demo. That's not the way forward, and an unnecessary limitation. We have VR. We have an audio-visual feedback loop. Computing resources available are *plenty* for convolving mono audio sources into 3D binaural output. It would be nice to have an opensource library using OpenCL to do this, since GPUs are fantastic for this operation. Nvidia, Intel, and others have been toying with this, even though binaural audio has been in need of an application for some time (waiting for VR to rise from the ashes!).

I've been working on something for quite a bit that uses Binaural beats, it's great to see a post on here regarding the use of them. I have it in the works in progress section for now until I release a build. (Invasion 2012). Sound will play a huge role in this game.