Unity: Floating gameobjects while walking

Recently I have been able to make a vr game using unity, however when I grab a gameobject and then walk around the gameobject tends to float around. How do I fix that?

I have tried parenting to the OVRrig and the Trackingspace, but gameobjects snap to the players forehead. I also tried to make a new transform point and they did the samefloating around bit. So right now they are parenting to the actual grip transform of the custom hands.
 You can see the problem in this video. https://www.facebook.com/brian.hillegeist.94/videos/586413755547290/