Rotating empty parent on character rotates on a weird axis when player isn't standing at origin

Sorry I know the title is really confusing.

Basically the camera and hands are nested under an empty object. This empty object is then used to rotate the character and teleport the character. Thing is, the camera is able to freely move around and away the empty object. Making rotation seem like you're running in a circle instead of just rotating. Then when you teleport it's a little offset. Like if you shoot the teleport beam straight down you move because it's teleporting the empty object to the camera which moves the camera.

So if the headset and camera aren't exactly on top of the empty object everything gets wonky. I understand the problem just can't figure out a solution.

I tried resetting the position on every teleport but you can still move away and spin in circles. As well as resetting resets the orientation and that's not exactly what I need. 

Any help would be greatly appreciated.