Passthrough doesn't work on client in multiplayer (made with Mirror in Unity)

Passthrough works fine on host player and shows him his room, but when connecting as a client the client's room is black. I am using Mirror in Unity 2021.2.7.

What might be the source of the problem? Is it possible to connect to someone in multiplayer and use passthrough at all?