Collision of Grabbable Objects

When I grab an object using OVR hands it does not collide anymore with any other collider, however when I release it, it collides. I want to know if I can make that object collide with other objects while being grabbed. For example, I want to grab a marker and write on a whiteboard within my scene. The board could be a cube with box collider.