How to make real-world objects appear in front of a virtual environment in Mixed Reality?
I am trying to create a Mixed Reality application in Unity where a virtual environment surrounds the user’s real room, so that the user feels like they are in a different environment while still being inside their real space.
I experimented with the SpaceMap sample from the MR Utility Kit:
https://github.com/oculus-samples/Unity-MRUtilityKitSample/tree/main/Assets/MRUKSamples/SpaceMap
This sample actually does almost exactly what I want.
However, I am facing the following issue:
Real-world objects, people, and even my own body appear behind the virtual environment objects, so they become invisible. In other words, the virtual environment fully occludes the real world.
What I would like to achieve instead is:
- Real-world objects, people, and my body should appear in front of the virtual environment based on depth.
- However, the room surfaces (walls, ceiling, and floor) should not occlude the virtual environment.
So effectively:
- Real dynamic objects should appear in front of the virtual world.
- Static room geometry should not block the virtual environment.
From my research, it seems possible to do something like this using opacity or passthrough techniques, but that’s not what I want. I would like the result to feel natural and realistic, without transparent effects.
Since this is my first MR application, I might be missing something obvious.
If anyone has experience with this or can point me in the right direction (depth APIs, occlusion techniques, or examples), I would really appreciate the help.