ScreenPointToRay and ViewportPointToRay not working with VR camera

X-posting from the Unity VR forums: https://forum.unity3d.com/threads/screenpointtoray-and-viewportpointtoray-not-working-with-vr.471440/

Environment:

- Unity 5.6

- VR is active and rendering to the HMD

My goal is to raycast from the mouse into the scene, with a VR camera.

When using ScreenPointToRay on a VR camera the ray's direction is way off. Here's an image when my cursor was in the center:

When using ViewportPointToRay the ray's direction is much better but still not perfect. Here's an image showing the difference (sorry for mouse pic, my screen capturer wasn't picking up the mouse):

Here's the test code to reproduce in a clean Unity 5.6 project and scene:

using UnityEngine;

public class Raycaster : MonoBehaviour
{
    public Camera sourceCamera;
    public bool useScreenPoint;
    public bool useViewport;
    public Transform viewportSphere;
    public float sphereDistance;
	
	private void Update () {
        Vector3 mousePos = Input.mousePosition;
	    Ray ray = new Ray();

        if (useScreenPoint) {
	        ray = sourceCamera.ScreenPointToRay(mousePos);
	    }
	    else if (useViewport) {
	        float normalWidth = mousePos.x / sourceCamera.pixelWidth;
	        float normalHeight = mousePos.y / sourceCamera.pixelHeight;
            Vector3 viewMousePos = new Vector3(normalWidth, normalHeight, 0);
	        ray = sourceCamera.ViewportPointToRay(viewMousePos);

	        if (viewportSphere != null) {
                viewportSphere.transform.position = ray.origin + ray.direction * sphereDistance;
            }
	    }
        Debug.DrawRay(ray.origin, ray.direction, Color.red);
    }
}

In these screenshots I'm using a single camera set to both eyes. However, I have repro'd the same problem when I switch to left and right eyes too (that's the setup in the actual project I'm working on).

To visualize the use case: there is one person in VR and one person not in VR but on the PC using the mouse to click on things the VR person is looking at. The simple solution to this would be to use another camera that displays just to the monitor, however the project is very performance restricted and the new camera would be displaying exactly the same thing as the existing VR camera.

Has anybody had to tackle this or have any ideas on how to solve this without an extra camera?