Ray Interactor raycast is detecting interactables behind other colliders

How can I prevent the interactor ray to detect interactables behind other objects? Scene objects do have colliders, for example walls, but I still can click on interactable object that are behind walls or other non interactive objects. I supposed that interactor ray raycast should be stopped by colliders, but it's not.

Is there a way to fix this without making a custom interactor?