Nested Grabbables

In my Project i want to have a floating shelf, with objects in it.
The shelf and objects should be grabbable. When I tried to develop such a system i placed the objects inside the shelf as children of the shelf in the unity hierarchy.

I can move the shelf and grab objects out of it, but as soon as i grabbed the first object inside the shelf it is not possible anymore to grab and move the shelf.
I use a color system where objects appear blue as soon as I hover over them, even that does not work anymore.

Blue is shelf, red are the objects inside.

Do you have aan idea ho to work with nested grabbables?