Unity can't move player back?

I'm having this super weird issue with the Oculus Quest 2.

In the editor and on the PC (With Oculus Link) this works perfectly fine.
What am I trying to do?
Cast a ray from the camera, if the camera gets to close to a wall move the origin back (which the camera is a child of) so the player can't move through walls. 

Now when I make an apk build for the Oculus Quest 2 this doesn't work at all.

Is there some android specific code that locks the TrackingOrigin in place?

The code I'm using isn't that special, any help here is appreciated! 

 if (Physics.Raycast(ray, out hitInfo))
        {
            if (hitInfo.distance <= 0.5f)
            {
                //If we hit something that is static
                if (hitInfo.transform.gameObject.isStatic)
                {
                    Vector3 movement = transform.forward * -1;
                    movement.y = 0;
                    //Move backwards
                    origin.transform.position += movement * Time.deltaTime * speed;
                    debugRayEnd.gameObject.GetComponent<MeshRenderer>().material.color = Color.green;
                }
                else
                {
                    if (hitInfo.transform.gameObject)
                    {
                        Debug.Log(hitInfo.transform.gameObject.name);
                        debugRayEnd.gameObject.GetComponent<MeshRenderer>().material.color = Color.yellow;

                    }
                }
                //  GetComponent<LineRenderer>().SetPosition(1, hitInfo.point);
            }
            debugRayEnd.transform.position = hitInfo.point;
            debugRayEnd.gameObject.GetComponent<MeshRenderer>().material.color = Color.red;



        }