Sleep recenters my camera

I'm making a game in Unity for Quest 2 and I have a problem with the camera.

If I'm positioned in a certain way in the game world and then takes of the headset long enought for it to go into sleep. When I put it on again a few seconds later everything is recentered in the direction I'm currently facing when putting it on. This is really frustrating if you make a game were you are in a cockpit and you have to recenter everytime if you take the headset off.

However. If I turn of the guardian system in the developer tab the world is NOT recenterd after sleep. Position, rotation and everything is remembered and is exacly how it was when I took the headset off. Is it possible to make it work likte this with guardian turn on too?

I'm very confused. Hope someone can help!

Thanks