MRUK scene loading inconsistently

I am working on a mixed reality game and using the MRUK to load scene data. "Load scene from device async" doesn't seem to work. Before using the MRUK, I used the OculusXrSceneActor, which worked wonderfully, but now im kind of lost. Now, using the "Spawn interior" function, it does work -- but only for my collegue. I'm not sure what is happening, could use any clue as to what is happening.

For some extra info: we check for permissions and launch the capture flow (which another post said could help fix issues with "missing" scene data)