Passthrough Over/Undlay switch at runtime bug

I try to use passthrough to fade between vortual and real space. Works theoretically, by dynmically adjusting passthrough opacity and switching from overlay to underlay passthrough.

However, if I switch passthrough to overlay and back to undelay, it stays in overlay mode and the virtual objects are not visible.

Any idea?