Inspiring

Missing Passthrough Scenario

4 replies
2078 views

TLDR: I want the quality of Reconstructed Passthrough with the flexibilty of Overlay and AddSurfaceGeometry, but I don't believe that is currently possible.

I've been exploring the capabilities of the Passthrough on Quest 3 using Oculus branch of Unreal 5.2. I've observed that Reconstructed Passthrough vs User Defined Passthrough has some unexpected differences, which I think comes down to "reconstruction artifacts" differences.

I've gotten the best passthrough experience using Reconstructed Passthrough, when I move my head or walk about my envirnment, there is minimal shifting and warping of the edge of my nearby desk. It seems very similar if not identical to the headset's home envirnment when passthrough is enabled.

The User Defined Passthrough in Overlay mode allows me to specify exactly where I want passthrough to show up in my scene by allowing me to specify the geometry that should be treated as a window. In my experiments, I've created a cube that surrounds the player, except for a hole in the cube where the player last recentered. This gives a "portal into the world" type effect that I am looking for. However, the pass through is noticably different. Comparing the corners of things with the home passthrough it is almost like it is using a different FOV. But the worst part is the moving my head around, it looks like the surface of my desk floats up and down and everything has a sickening floating feel to it. I wonder why this is happening, and I suspect that "user defined passthrough" is also "surface projected passthrough" and so I suspect that the projectioning is happening differently.

What I really want is to have the flexibility of defining where passthrough will show up (like "User Defined Passthrough" gives me) but I want it to use Reconstructed Passthrough for these areas so that it matches the experience on the Quest 3 home screen.

I don't know how it works. After writing the above, I've spent the day reviewing the relevant code under OculusXRHMD\Private. I can only understand so much because the OculusXRPlugin's external lib obfuscates my ability to understand this further and I could not find the source for that in github. I would have thought that there would just be a single passthrough experience, and my app would just be poking holes that expose that underlying experience. If that is the case, then this may be a relatively simply feature to support, and they just didn't think that users would want it. I'm worried that isn't the case though, and that they are trying to save work creating the whole reconstructed passthrough when it isn't fully needed and instead built this reprojection system onto custom geometry as a performance feature. I would gladly pay the cost of having full passthrough running if it meant my passthrough wouldn't make my users sick.

I would be interested in hearing if anyone agrees that what I want simply isn't possible, or if any thinks it might be possible. Also, any thoughts or insights anyone has I would be interested in hearing them.

This topic has been closed for replies.

K

kampfz

Participant

did you ever make any progress on this? i am trying to be able to create portals into a virtual space (similar to the quest 3 demo game) with unreal.

I thought the pokeahole material would be useful for doing this but it does seem like this functionality is missing from the plugin, unless I'm missing something?

CarpetFaceAuthor

Inspiring

I have not made any progress.

I understand your confusion; it appears that a lot of the documentation was written with the Quest Pro in mind and was written before Quest 3, and things have straight up just changed without the standard deprecation period that most of us are typically expect. The closest thing to "pokeahole" is "User Defined Passthrough in Overlay mode".

But as I mentioned above, I was not happy with this solution because it does a reprojection of what the camera sees onto this geometry, and the geometry slightly warps things to the point that moving the HMD results in vomit inducing shifting of the world. I believe the Reconstructed Passthrough looks so good because it is using the room geometry and doing the camera projecting onto that geometry, which gives us the quality that we've come to expect and love. But there does not appear to be a way to "pokeahole" in one's seen so that we can see the Reconstructed Passthrough.

If you want to match the Quest 3 demo game, you can still achieve this by using Reconstructed Passthrough in underlay mode, and then drawing your world on top of it, which turns the problem into "How do I render a portal in Unreal with Vulcan forward rendering?" for which I do not know how to do it. I suspect the demo was written in Unity.

If my understanding of the problem space is correct, this means one won't be able to create a "portal" into your game's "virtual world" in the middle of the player's play space and have that player step inside, and then have it behave like they are inside it, because managing the transition from pass through with portal, to stepping half way through and having one eye see "from the other side of the portal" while the other eye is still outside of the portal, won't be possible, and will require a complex and non-visually-ideal solution. You may not be attempting this route, but I was and that is why I gave up (Note: I got close to exactly what I wanted, but the shifting of the pass through with user defined geometry does NOT feel good unless that user defined geometry closely matches the location of the real world objects that are being projected onto them).

K

kampfz

Participant

I just came across the Mixed Reality Utility Kit. I'm thinking this could be the key to helping me achieve the effect that I'm after. Looks pretty powerful

V

vrgroupAachen

New Participant

Has anyone made any progress on this? I can't really find any information about potential ways to do it nicely.

Get Help

Sign up

Login with SSO

Login to the community

Login with SSO