Oculus Rift framerate randomly becomes stuck at 45 or 30 FPS when headset presence changes

Running
into an odd problem here with a Unity project. I'm only using 4 ms of
CPU per Update and not doing anything taxing on the graphics side. It
effortlessly hits 90 FPS... most of the time.

Sometimes (about 3/20 times) when I remove the headset and put it
back on the framerate drops to 45 or 30 FPS. The problem usually goes
away if I poke my finger in front of the head presence sensor a couple
times. According to the Unity profiler this problem is all vsync, which
makes me suspect something odd is happening on the Rift's side of
things.

Any ideas on how to fix this?