OVRCameraRig position vs CenterEye position

So im using as input for movement in my game the x and y position of the CenterEyeAnchor, given the object z movement being variable with no input. This makes my object flow in front of the camera with the head movement thanks to occulus.

This works great, but now i wish for the Camera to follow the x of the object aswell. To do that i need to work with the x position of the OVRCameraRig and doing that im also moving the x CenterEyeAnchor's position which makes a ridiculous speeding effect. Is there anyhow to isolate one movement from another? or an alternative for this?