OVRGazepointer with disabled position tracking

The OVRGazepointer does not work well when the position tracking is disabled in the camera rig. The cursor is no longer centered, sometimes don't even follow the camera and the UI click areas have some form of offset. I have to move further downwards to click on a button. I'm using rift for my testing and need to disable head tracking and select with the pointer.