Forum Discussion
When there's no need to render twice
Conventional Rift engine design currently appears to focus on rendering the entire scene twice, once from each eye. The only alternative that seems to get discussed is stereo reprojection using a depth buffer, but that's universally agreed to be unsatisfactory.
But actually, there's no need to render your entire scene twice in every circumstance. Thanks to the Rift's display's low angular resolution, there's a fairly low distance threshold beyond which stereoscopic depth cues are no longer distinguishable. We can calculate this threshold distance thus:
Additionally, as FredzL pointed out in IRC, distance stereopsis is not a primary depth cue at longer distances:
One technique to take advantage of stereo imagery being unnecessary beyond a given range would be a three-pass renderer:
The downside to this technique is that you're losing the benefit of the z-buffer cull, having to overdraw parts of your scene. However, this could be compensated in part by occlusion culling routines of other sorts - including portaling. For indoor scenes there would not be much benefit to this, however, as in most scenes that work well with portaling engines you've already got the vast majority of your rendered content in the foreground.
However, for scenes such as flight or space simulators (where your cockpit is the only thing within ~50m of you in normal gameplay, and therefore the only portion that needs to be rendered in stereo) this technique could make a huge difference to performance.
An alternative render order could be:
[list=2]
Passes 1 and 2 render left and right eye scenery with a far clipping plane of 60m, with the depth buffer rendered to an external render target for each eye
Pass 3 renders centre eye scenery with a near clipping plane of 60m, testing against a depth buffer written out from each eye in pass 1 and 2
Barrel distortion shader etc
I've not really thought much about this method yet, since it requires some way of substituting for having two independent z buffers. One way could be using the alpha buffer of the texture as a depth buffer for the depth test - but this would require some shader fiddling, and you lose transparency.
What I'd like to know is - has anyone already tried to implement a similar technique, and if so, what did you discover? I find it hard to believe I could be the first person to try to implement such an obvious idea.
But actually, there's no need to render your entire scene twice in every circumstance. Thanks to the Rift's display's low angular resolution, there's a fairly low distance threshold beyond which stereoscopic depth cues are no longer distinguishable. We can calculate this threshold distance thus:
- Average interpupillary distance (IPD) ~= 0.064m
- Best angular resolution = 1280px horizontal resolution / 90 degrees horizontal view angle = 14.22 pixels per degree
- Smallest angular size of a pixel = 1 / 14.22 PPD = 0.07°
- ∴ Distance at which stereoscopic effects are smaller than 1 pixel = 0.064 / tan(0.07°) ~= 52m
Additionally, as FredzL pointed out in IRC, distance stereopsis is not a primary depth cue at longer distances:
One technique to take advantage of stereo imagery being unnecessary beyond a given range would be a three-pass renderer:
- Pass 1 renders the background from the centre camera with a near clipping plane of 60m, onto both eye render target textures
- Pass 2 renders the left eye view from the left eye camera, with a far clipping plane of 60m, overlaid onto the left eye target texture
- Pass 3 does the same for the right eye
- Barrel distortion shader etc
The downside to this technique is that you're losing the benefit of the z-buffer cull, having to overdraw parts of your scene. However, this could be compensated in part by occlusion culling routines of other sorts - including portaling. For indoor scenes there would not be much benefit to this, however, as in most scenes that work well with portaling engines you've already got the vast majority of your rendered content in the foreground.
However, for scenes such as flight or space simulators (where your cockpit is the only thing within ~50m of you in normal gameplay, and therefore the only portion that needs to be rendered in stereo) this technique could make a huge difference to performance.
An alternative render order could be:
[list=2]
I've not really thought much about this method yet, since it requires some way of substituting for having two independent z buffers. One way could be using the alpha buffer of the texture as a depth buffer for the depth test - but this would require some shader fiddling, and you lose transparency.
What I'd like to know is - has anyone already tried to implement a similar technique, and if so, what did you discover? I find it hard to believe I could be the first person to try to implement such an obvious idea.
11 Replies
- I haven't tried this or anything, but it seems like a really good idea. Sort of like a dynamic skybox. It could be tricky in some games to make sure that far away things never end up being close, and vice versa. I imagine the transition between far and near cameras would be pretty nasty.
- Edit: removed since it's bullshit and wrong.
- It could possibly work. You will need to be careful of the seam between the shared far frustum and the separate near frustums - you could get single-pixel cracking as objects cross that boundary. It may be that by shearing the near frustums you can make them meet in a mathematical sense, though I'm not sure how visible that shear might be in the geometry as it gets close to your eye - human brains tend to be sensitive to those sort of things.
But even then making them meet mathematically may not be enough given that the math is being done with finite-precision floating point, and you may still get cracks. So still worth trying, but yes - could get tricky. Good luck! "dghost" wrote:
The problem that I see with it is that it doesn't take into account what's happening with the frustum, and primarily in two ways:- The primary source of stereoscopic effect comes from the asymmetric view frustums, with the point of minimal parallax occurring at infinity. Simply rendering the scene once and trying to reuse it will cause objects to have improper parallax and not be able to converge correctly. Given that the mathematical point of minimal/indistinguishable parallax is infinity, while I do believe an equivalent value can be calculated for the view frustum (it would basically be the range in the depth buffer that is indistinguishable from infinity given your current screen)
Thanks for the input, but i'm afraid you might have totally missed my point - of course it's true that mathematically parallax connects at infinity, but in the Rift (or on any raster display device), there is a limit to distinguishability due to its resolution. The whole point of my post was to calculate that limit, and from it infer a distance beyond which there is no distinguishable parallax between either eye. With the rift's low res, that's just 53m away from the camera."dghost" wrote:
- Naively adjusting the clip planes of the view frustum will also mess with parallax. Given that the near clip plane always maps to the same value in the depth buffer, and the far clip plane always maps to what is essentially infinity in the projection, rendering different areas of the scene with different frustums would cause depth violations to crop up.
We're not talking about rendering different areas of the scene with different frustums and keeping the same depth buffer; that would make no sense at all, and of course the results would be junk. We're talking about a full render of different depth slices of the same scene; in each case a fresh depth buffer is used, because there's absolutely no point trying to do depth tests against a slice we *know* has no overlap. Every pixel in the two near frustums will by definition always be nearer than any possible pixel in the far frustum."tomf" wrote:
You will need to be careful of the seam between the shared far frustum and the separate near frustums - you could get single-pixel cracking as objects cross that boundary. It may be that by shearing the near frustums you can make them meet in a mathematical sense, though I'm not sure how visible that shear might be in the geometry as it gets close to your eye - human brains tend to be sensitive to those sort of things.
Well, the point is that geometry wouldn't get close to your eye - the shear point would be behind the distinct resolution boundary, at least 50m away from the viewer. I agree that getting the seam right is critical though. I do think that the devkit 1's low resolution will go a long way to masking any artifacts at the boundary anyway."tomf" wrote:
But even then making them meet mathematically may not be enough given that the math is being done with finite-precision floating point, and you may still get cracks.
Z-fighting for deep scenes is a major problem caused by floating point imprecision, granted, but actually this technique should reduce that problem for two reasons:
1. You've got a lot more z-buffer precision, since you're rendering the scene in two depth passes rather than one.
2. If you overdraw the background rather than trying to combine in a mutual depth buffer, then there's no possibility of any background popping in front of foreground at the seam. As a result, you can actually round down the frustrum starting depth of the background to the nearest floating point number *before* the boundary, and overdraw from there - that should allow for a bit of seam "glue" to fill any possible gaps.
Still, i'll need to build a prototype before i can confirm it would really work, in a perceptual sense. Will aim to do that in the coming weeks when i have some free time :)
- The primary source of stereoscopic effect comes from the asymmetric view frustums, with the point of minimal parallax occurring at infinity. Simply rendering the scene once and trying to reuse it will cause objects to have improper parallax and not be able to converge correctly. Given that the mathematical point of minimal/indistinguishable parallax is infinity, while I do believe an equivalent value can be calculated for the view frustum (it would basically be the range in the depth buffer that is indistinguishable from infinity given your current screen)
- Apologies, I do actually get your point. Unfortunately, I was bungling the math a bit when it came to the projection matrix. For whatever reason I thought the projection offset varied based on the depth, although I did work through the math again and can clearly see that it does not. Same goes for my concern about depth violations - for some reason I had it in my head that the projection offset was not constantly applied and using two different sets of clip planes would screw with parallaxing. So again, apologies for that.
It is an interesting idea. I would wonder how this would fair when dealing with higher-than-native frame buffer resolutions - my guess is your math has plenty of breathing room since you have significantly less than 1280px visible per eye, but if it did it's trivial to recalculate it based on the frame buffer resolution instead.
I would also be curious if it causes any artifacting at the boundaries from interpolation/rasterization differences for texture coordinates and vertex edges. Even if the offset caused by stereoscopic separation is below the pixel threshold, it's somewhat hard to predict how that could subtly impact rasterization. - i have just 1 thing to say...checkout a game called secondlife - specifically the sourcecode :)
- Did you ever create a prototype of this? Seems like a pretty good idea to me..
By the way, your math is a little off, the Oculus Rift renders at 640x800 per eye, not 1280x800. So it's more like this;- Average interpupillary distance (IPD) ~= 0.064m
- Vertical angular resolution = 640px horizontal resolution / 90 degrees horizontal view angle = 7.11 pixels per degree
- Horizontal angular resolution = 800px vertical resolution / 110 degrees vertical view angle = 7.27 pixels per degree
- Smallest angular size of a pixel = 1 / 7.11 PPD = 0.140625°
- ∴ Distance at which stereoscopic effects are smaller than 1 pixel = 0.064 / tan(0.140625°) ~= 26.08m
And for the HD Rift prototype, it's 960x1080 per eye (with the same field of view?) which works out to ~36m - Actually the horizontal FOV seems to be a bit higher than 90°, more around 114.5° considering this post (default horizontal FOV from the stereoconfig object in the SDK).
Also the horizontal angular resolution is not linear, it would be 640/114.5 = 5.59 in average but it should be higher in the center FOV because of the pre-warping.
And although the display is using discrete pixels, sub-pixel resolution can still be achieved through aliasing.
As illustrated by this image, we can discern the subtle movement of the right arm although its amplitude is only around 1 or 2 pixels horizontally :
In this one a sphere is rendered at sub-pixel resolution (top) and discrete resolution (bottom) :
But the idea is still interesting since stereopsis is no longer a primary cue past a certain distance as the OP said (around ~65m according to what I've read in several papers). It would be nice to see it implemented to evaluate where the best compromise can be found between gain in performance and loss in depth perception. - Well, you should then replace "less than one pixel" to "less than one sample" and it's working again. But, just to ensure smoothness, multiply this by two (less than half the sample). So I guess with 16x MSAA/SSAA you can render to a mono-buffer objects that are further away than ~200m. So if you game have a lot of large open spaces, it could really benefit from this. But as the resolution increases, this benefit diminishes even further.
"SlowRiot" wrote:
An alternative render order could be:
[list=2]- Passes 1 and 2 render left and right eye scenery with a far clipping plane of 60m, with the depth buffer rendered to an external render target for each eye
- Pass 3 renders centre eye scenery with a near clipping plane of 60m, testing against a depth buffer written out from each eye in pass 1 and 2
- Barrel distortion shader etc
I've not really thought much about this method yet, since it requires some way of substituting for having two independent z buffers. One way could be using the alpha buffer of the texture as a depth buffer for the depth test - but this would require some shader fiddling, and you lose transparency.
This could be done with a single depth buffer by taking the minimum depth between the left and right image (since the culling should be similar, and then just draw the left and right eye over the center eye scenery, since it is garunteed (not sure about shadows, effects, etc.) to be nearer to the center eye scenery."Fredz" wrote:
And although the display is using discrete pixels, sub-pixel resolution can still be achieved through aliasing.
AFAIK, sub-pixel rendering requires either super sampling (which only changes the near-far clipping threshold) or blurring (which should still behave identically).
