Active Ray Interactor switching between interactables

Heyo I'm using SDK v60,

in my scene I have a floor that consists out of multiple seperate tiles. Every tile has their own Ray Interactable component and I'm casting the ray from my hands. The cursors are correctly displaying on the surface, switching seamless between tiles.

Now when I make the Pinch gesture, it locks the cursor and CollisionInfo to that specific selected interactable. That means everything stops updating when I leave the border of the selected interactable. I guess thats expected behaviour, but is not helpful when I'm trying to paint something for example and don't want to release the pinch every time, before switching to the next tile.

I would like to implement a seamless switching between interactables, while keeping the pinch active. I tried Unselect(), without results. Maybe somenone here give me some pointers.