Extremely slow performance from "GetSpectrumData" of an AudioSource on Quest3 (likely 2 as well)

I'm developing a music visualizer and have been seeing some odd behavior when accessing the spectrum data from a playing audio source on a Quest3.  As a test, I have some cubes reacting to each of the audio bands coming in from GetSpectrumData (in this case, 512).  In the unity editor on PC it works great and matches the framerate perfectly.  On-Device though, the application seems to be running at a smooth 90hz, however the spectrum data is only updating at what looks like 5-10hz, possibly less. I have manually set the audio sample rate of the application to 48000hz but that doesn't seem to help.  Has anyone else run into this or have any advice on what I can try to address this issue?

EDIT: Doubling the rate to 96000hz helps a bit but it's still choppier than i'd like, and i'm a little worried that the onboard DSP isn't going to be too happy running running at that rate for a long time.

EDIT2: I set the application's framerate to 120hz as a test and i am seeing the same results.  The application is running at a locked and incredibly smooth 120hz (verified with graphy) while the audio data is refreshing only a handful of times per second.  96000hz appears to be the highest that unity will let me set (fmod starts to complain if you go higher) and while that is.....fine.....it's still nowhere near as smooth as it is when running in the editor.  So, this doesn't appear to be a performance limitation since the app is staying at 120hz, but a limitation somewhere else.  I'm open to any suggestions that anyone has.