Anyone got OVRGrabber working with a Remote Avatar?

Trying to show a recorded avatar pick up an object - unfortunately a recording of the avatar does not sync with a gameobject record of the picked up object. I'm sure there's a way to hook up the OVR Grabber to the remote avatar, but can't quite figure it out. Anybody have any ideas?