Grabbing Objects with Multiple Rigidbodies in Unity

Hi folks,

I am having trouble figuring out the best way to handle this situation. I am using Rift and the OCVRgrabber/grabbable scripts. In the game, the player is going to assemble an object. The “main object” has a rigidbody. Its child gameobjects are just colliders. I have had success allowing the players to attach pieces. The pieces have rigidbodies. When I attach the piece, I make it a child to the “main object”. 

I am having to remove the piece’s rigidbody because setting it to kinematic does not work. The piece seems to react to gravity or cause the main object to float in midair. I suspect that when the grabber grabs the main object (changing it to kinematic), it affects the rigidbody of the child. 

I need the rigidbody to still be attached, because I want to be able to remove the piece later. I will need to know what piece my grabber is colliding with to be able to pull it off.

Is there a recommended way of having a grabbable object that has multiple rigidbodies attached? Or should I be thinking about fixed joints or a script to keep the items together without parenting?