Specular / Reflection mismatch in stereo!!

Can anyone at Oculus or Unity confirm or describe what this issue is? Or how to solve it?

Hi Infinite, I tried to watch the video you linked but got a "Permission Denied. Sorry, there is no video here." message. So I may or may not know what you're referring to.

If you use a simplistic shader that does reflections based on a single reflection cubemap, the reflection is purely dependent on your overall view direction (same for both eyes) and thus you perceive a reflection that is off at infinite distance. The stock reflective shaders in Unity will give you this effect and it doesn't look right in VR. There's plenty of reflective shaders out there that do this right though.

There's also the matter of specular/rim/fresnel highlights that don't feel right, and looks a little different for each eye than your brain is expecting. I see this a lot in Senza Peso (on the rock walls in one of the first couple of parts, I forget which exactly). If the shaders recalculate the highlight for each eye, then it should look better, but this also depends on whether or not you're using a single reflection cubemap for your reflective colors.

I'm probably barking up the wrong tree though, huh?

Hi Drash,

Thank you kindly for the reply.

What you mention is exactly the issue but I wonder if this is a shader implementation issue, an engine issue, or SDK issue? Who can solve it? As Marmoset Co highlighted, they are aware of it but don't seem to know what to do about it, as it hasn't been fixed in the latest Toolbag 2.0 release. So I wonder if it's a core issue with stereo rendering?

I wish I could find John Carmack's tweet on the phenomena.

The problem with conventional environment maps is that they are effectively reflecting light from infinitely far away. That's fine for the sky, but for closer objects it looks obviously wrong.

Parallax corrected cube maps are one fair alternative, and Unreal 4 uses them extensively. Specular from point light sources should also be correct.

The ultimate answer is we need true real time global illumination to give us correct per-pixel reflections.

"Infinite" wrote:
I wonder if this is a shader implementation issue, an engine issue, or SDK issue?

In most cases, it is a shader issue. The game needs to render the reflected image with the correct projection. That usually means an off-axis frustum. Unity's mirror shader seemes to do it right, but Shyshop might be doing something else.

"owenwp" wrote:
The problem with conventional environment maps is that they are effectively reflecting light from infinitely far away. That's fine for the sky, but for closer objects it looks obviously wrong.

Parallax corrected cube maps are one fair alternative, and Unreal 4 uses them extensively. Specular from point light sources should also be correct.

The ultimate answer is we need true real time global illumination to give us correct per-pixel reflections.

Thanks. "Parallax corrected cube maps are one fair alternative" this might be worth exploring further.

"vrdaveb" wrote:

"Infinite" wrote:
I wonder if this is a shader implementation issue, an engine issue, or SDK issue?

In most cases, it is a shader issue. The game needs to render the reflected image with the correct projection. That usually means an off-axis frustum. Unity's mirror shader seemes to do it right, but Shyshop might be doing something else.

Thanks for the reply. Would you be able to link to an example of the mirror shader you mentioned? Hopefully someone form Marmoset might be able to comment further. It would be amazing to find a fix for this issue.

To further add to the discussion some of the Marmoset Co guys have responded by email and would be interested to talk with Oculus or Unity further about this via Twitter, if the interest should arise.

I think this is a very important problem to solve and could have a profound impact on shader realism in VR:

I'm wondering if you've tried Skyshop's parallax corrected reflections (box projected skies) and if they help at all.

Also, as of Skyshop 1.08 or 09, I've converted to computing the eye vector as _WorldSpaceCameraPos - worldPos instead of relying on Unity's viewDir parameter. If this has fixed reflections, viewDir derivation in Unity's shader system is to blame. If this does not fix it, I'm willing to bet cameraPos is not configured properly for the two eyes.

If someone from Unity or Oculus could tell us how the two cameras are derived and which matrix the eye split is stored in, we'd have more clues to go off of.

Andres Reinot
Engineer - Marmoset

Twitter handle - @monkeyscience8

https://twitter.com/monkeyscience8

---------------------------------------------

Ah yes, this old problem. Andres and I spent a bit of time discussing it this morning. The conclusion is that we still aren't sure :P I can I think rule out a few things though, and I have one more guess.

One is that it's not an issue of parallax correcting the cube map (or rather, that's a different problem). Surfaces should be able to be rendered with infinite reflections without the painful artifacts we've seen. Proper parallax correction of cube environment maps would be a nice improvement for local reflections, but I don't think that's what we're dealing with here.

I think I can also fairly safely rule out some trivial bugs on our part regarding off-axis frustums or the like - we do very carefully use the correct camera positions from the SDK, as do many other apps, and the problem still remains in all of them. If you're using Skyshop in unity specifically, and there's some discrepancy between the unity shaders and ours, then there's likely a bug, and we'd like to hear more, as Andres says.

I have a reflective sphere on my desk, because I'm that kind of person. When I pick it up and hold it at arms length, and alternately close each eye, I can't tell much difference between the reflected images. If I move it much closer to my nose however and do the same, I can tell a large difference between each eye. With both eyes open its not possible for me to converge on features reflecting near the edges of the sphere (where there is the most distortion). This, qualitatively, is the same sensation I get with the headset on. Try it!

So, a new theory:

It's hard to resolve reflections when the object size/distance is comparable to your eye separation distance. This is because the distorted reflections vary so drastically from one eye to the other that your brain can't quite resolve it as a clear image. So I think one has to be careful noting the eye separation distance vs the distance to the subject - since one doesn't have focal cues with the headset on you have to be careful that you aren't "holding your subject against your nose" so to speak, as reflections are more sensitive than general parallax is to this effect.

Hope that's helpful,

Jeff Russell
Engineer, Marmoset

Twitter handle - @j3ffdr

https://twitter.com/j3ffdr


---------------------------------------------

I hope this will further the discussion to resolve the problem.

"Infinite" wrote:
Would you be able to link to an example of the mirror shader you mentioned?

Here is one from Aras: http://wiki.unity3d.com/index.php?title=MirrorReflection2

Thanks for the link but it's a little confusing as that mirror effect is done via a script (doing all necessary calculations for the camera) rather than a shader, the shader just consists of pass applied to it for the reflection.

I hope the discussion might continue further with Oculus, Andres and Jeff via Twitter.

Having the ability to correctly converge or focus the reflection and specular highlights in stereo will look incredible in VR, at the moment it's headache inducing! and really breaks immersion.