Camera Rig Moving upwards with collisions

Hello everyone, I'm currently working on a Unity VR Game using the Meta SDK (formerly Oculus VR). In my game, you grab different objects and manipulate them. The Scene is mainly a table collider with multiple rigidbody Grabbable Objects on it made from the sample ChessPiece prefab. I'm using the base Camera Rig and Interactions Building Blocks.

 I'm encountering the following issue : When grabbing objects and moving them, the camera rig parent GameObject sometimes gets moved up, permanently offseting it on the y axis. I think this may have to do with the physics system, but I can't figure it out. 

Do you have any ideas what might cause this bug ?
Thanks in advance, Joseph