create a gameobject allowing touch but preventing penetration

I use Meta XR All-in-One SDK. I want to achieve that even if the real world hand moves, the virtual world hand still cannot pass through the gameobject we created. We can use the hand to touch the surface of gameobject, but cannot pass through gameobject. 

I try to set up virtual hand with colliders use CapsuleCollider and Rigidbody. And attach to "XRHand_IndexProximal", "XRHand_IndexIntermediate", "XRHand_IndexDistal", "XRHand_LittleProximal", "XRHand_LittleIntermediate", "XRHand_LittleDistal", "XRHand_MiddleProximal", "XRHand_MiddleIntermediate", "XRHand_MiddleDistal", "XRHand_RingProximal", "XRHand_RingIntermediate", "XRHand_RingDistal", "XRHand_ThumbProximal", "XRHand_ThumbDistal", "XRHand_Palm".

Even if I set up the collider, the hand will not stop moving just because of the collider. I have tried it so far, and I can only use collider to push objects, but I can't make the object and hand not move when the hand pushes the object.