raycast on player controller hands not working

So I set up a script on each hand (OVRPlayerController>OVRCameraRig>TrackingSpace>Left-/RightHandAnchor) with a raycast and a line renderer showing that raycast so that you can click objects in the scene.

It works find for the most part, except that it only seems to actually do the raycast when the HandTrigger button on the oculus controller is being held OR if I'm moving the controllers pretty fast. It's like it's not detecting anything or not doing the raycast unless I do those things. I've searched through my entire code, and there's no stray place where I'm taking extra input. The only thing I can think of is that some part of the oculus code is trying to be more effiicient by not calling the raycast as often as I want it to.

Any ideas?