Camera facing problems

In my scene I have a character that faces a tree. I have set the ovrcamerarig prefab at the position of the character's eyes. The character model has an empty gameobject as parent. I move and rotate the empty gameobject to move and rotate my model. My ovrcamerarig has the model's head as parent. I suppose that's not necessary, but I do need to have it as a child of the empty gameobject.

The problem is that when I have my head turned the moment the scene loads, the game thinks i'm actually looking forward (so i'm looking at the tree). So if I then look forward again, the ingame camera rotates to the right so that i'm looking at the character's right arm.

The way it is now, if the player is looking around the moment a new scene starts loading, the camera and body will not face the same direction.

I'm sure it must be very simple to fix this as what I want isn't that uncommon, but I haven't found a solution yet :s