Low poly-count indoor scene with performance issues while looking outside the room (weird pattern)

I think I'm really stupid.

I believe all the problems related to that is sensor misconfiguration and head turning back without a third sensor to help. I don't know why, but the default setup in Unity is not as great as what was being done in Unreal. The default VR template in Unreal doesn't give as bad as the result in Unity doing the head movement pattern that I've shown in the clips above. I've setup a simple scene in Unity and the problems happened the same way.

I think I'll go with me being the issue here.