VR Mixed Reality Passthrough + Multiplayer: What determines "Forward?"

For OVR passthrough, what determines the worldspace "forward" direction? I'm working on a mixed reality multiplayer app and I need to, therefore, have all headsets calibrated to the same worldspace origin and forward direction.
I've calibrated the same space on both headsets, even setting up the walls in the exact same order each time. Even then, one headset sets forward as towards my wall, the other towards my desk.

In the included screenshot, the red line shows where universal forward is pointing. I want it to face away from my desk, not the wall. I can't seem to change it under my own control and is determined at roomscale setup?