It depends on how many spatialized sources you are playing and if those sources are going through early reflections. CPU usage can add up very quickly if you are not careful; for example, larger room sizes with reverberation on will almost certainly saturate the audio CPU when you have a lot of voices.
I would try turning early reflections off globally (found in the mixer bus plugin) and see if that reduces the audio CPU usage. Profiling audio will give you a good idea of how much the CPU is being used in your game.
btw I recommend sticking to the release versions of Unity. I am hearing some issues with 5.40b4 in the test scene when using best latency, but this does not occur in 5.3.1p3 (the most recent public release I am using).
