Multiple OVRCameraRigs

I have a Scene with 3 objects, each of which contains a bunch of stuff and it's own camera. Originally I just had 3 cameras, no OVR scripts on them, and when I turn far enough I disable one of the objects and enable another one, so when you turn past a certain point it switches scenes. This was all working fine until I wanted to add Touch controls.

I added OVRManager to an object that is always on, and each camera got an OVRCameraRig. If they each have their own TrackingSpace, or if I move the TrackingSpace on switch, the forward direction of the controllers is off by 120 degrees (the amount I rotated). I want to either totally reset the tracking when I switch, or move over my TrackingSpace and reset the forward direction to line it up with the new camera. I've gotten pretty close, by moving the tracking space and then disabling/enabling OVRManager with ResetTrackerOnLoad checked. The controllers work, they are in roughly the right place, but they move along with my head when they shouldn't.

Is there a better way to do this that still let's me keep the separate cameras? The cameras all have different scripts/settings etc. so I don't want to use a single one if I can avoid it.