Screen coverage - why isn't it maximized?

Instead of rendering a normal image and warping it, is it not possible to implement an in-game lens shader that produces the FOV, dimensions and curvature natively? This can be done with software packages like Mental Ray, VRay, Maxwell etc, so I imagine it must be possible within a game engine.

Erm, no?

I take it you don't know what a lens shader is.

Here's one for Mental Ray, which renders a 360 degree image.
http://www.lvyuedong.com/projects/tech/mrPanoramicStereo/mr_panoramic_stereo.html



Because the image is actually rendered that way, there's no need for post image warping. Why not integrate something like this into Oculus games? You'd no longer need to supersample and distort a standard image, and the game could naturally fill the entire Oculus screen from edge to edge.

"Pogo" wrote:

But I still think curved screens should be implemented with the hardware

Oh absolutely, I agree 100%. Slightly different topic, but yes. OLED can be shaped - therefore it would make sense to shape two panels to the curvature of each eye, presumably consuming much more of your left/right vision.

OLED screens can be curved in one dimension, like bending a sheet of paper. However, curving them in two dimensions is a much harder engineering problem. Further, bending the screens in any fashion doesn't really improve anything. I've written something of a rant on this topic here: http://rifty-business.blogspot.com/2014/01/flexible-displays-rift.html

"Pogo" wrote:
However, I'm not sure if concave lenses like the ones in the Rift would work with curved displays. I get the feeling that, because they're designed to "bend" the image around your vision, that the effect would be double if not more with a concave display. As we know, the lenses are needed for focus, so I'm not sure how you'd combat that.

The lenses are convex, not concave. Concave lenses curve inward, creating a sort of hollow or 'cave', which is the mnemonic I use to remember the difference. The lenses do two things simultaneously, they produce a collimated light view of the display panel, making appear in focus, as if it were infinitely far away and increased the perceived field of view. The practical effect of this is that the light hitting the eye from a given angle goes to a specific pixel, regardless of where the eye is inside the 'eye box' a small region in front of each lens that produces a clear view of a sufficient amount of the screen. This is why the lenses themselves don't need to be adjusted to match the users IPD. The lens axes are about 6cm apart, but as long as the users eyes are somewhere in the range of 4-8 cm apart, the view they get in each eye is the same as anyone else's.

It's probably possible to make lenses that would work with a bowl shaped display. But a bowl shaped display panel would represent a huge engineering cost and require display panels that were specific to Rift hardware rather than relatively cheap commodity hardware (even a low persistence display made mostly custom for the Rift will still be using most of the manufacturing pipeline used for tablet displays). Bowl shaped panels would easily make the Rift cost an order of magnitude more and provide no actual benefit to the user, since there's no way to remove the need for lenses.

"Pogo" wrote:
I'm just questioning why it seems a lot of developers aren't rendering to the full extent of the Rift panel because that does in fact undermine one of the major initiatives of the project.

Yeah, that's a curious question. I'm working on another reply to the OP talking about it.

"Pogo" wrote:
My question is, if the horizontal FOV of the Rift is the most limited, why are the stereo images we see in so many demos taller than they are wide?

Part of this is simple geometry. The aspect ratio of the per-eye view of the panel is 4:5, so there's about 25% more vertical space to work with than horizontal space. That means that if a developer is working with an offscreen framebuffer with the same aspect ratio as the target viewport (which is typically the case as it makes calculations easier) then you'll always end up with more rendered vertical FOV than horizontal.

Essentially the issue is that people are choosing (or are somehow constrained) to render to a smaller FOV, and that results in the smaller FOV on both the horizontal and vertical scales, but since the vertical scale is already larger, the loss of FOV becomes apparent in the horizontal directions first, unless the developer were to do extra work to make the rendered scene have a 1:1 aspect ratio.

"Pogo" wrote:
I see unused screen space around the stereo images on many screenshots and demos I've tried. Surely they should extend to the outer most edges of your screen?

This is indeed a curious question. I've already opened a bug against the Steam VR API here after going through it and discovering that using this API drastically reduces the field of view and provides no mechanism to increase it.

There are a number of reasons that this might be the case.

Adapter constraints

I don't have a detailed understanding of how the tools like vorpx work, but adding Rift support to a game that wasn't designed to have it natively probably means that there are something you just can't do. If vorpx has no way to force the engine to render a higher FOV or to alter the engine's projection matrix on the fly, then you're going to lose something when you end up putting it on the Rift.

Engine constraints

Some gaming engines simply may not be designed to have a customizable FOV, or may have limits on how far it can be extended before you start to get negative artifacts. Perhaps for optimization reasons a game assumes that outside a given maximum view frustum objects don't need to be rendered and drops them early in the rendering pipeline. The amount of work to fix that might be high enough that it's not worth doing.

Performance constraints

It's important to remember that you have to actually render at a significantly higher resolution than the actual display panel. Because the shader has a magnification effect near the center of the screen if you render to an offscreen buffer that's equal in size to the per-eye resolution (in this case, 640x800) then the pixels at the center of the screen will be somewhat blurry. I'm working on an example application that demonstrates this. Regardless, even though the display panel per eye is only 640x800, in order to get no blurring in the center of the image due to magnification, you'd have to render to a framebuffer resolution of about 1100x1400. And you'd have to do it twice per frame in order to render to the Rift.

If you look at the implementation of the Valve Steam VR API here you'll see they are setting something called the distortion fit point to a value of 0.6 (horizontially) and 0.0 (vertically). This implicitly modifies the field of view and is essentially saying to the Oculus SDK 'I want to render an image that will only reach to 60% of the way to the edge screen as measured from the center of the screen. Since you're rendering to a smaller area the distortion effect has less magnification and the resolution of the frame buffer can be lower, in this case about 770x960.

Depending on the kind of game you're running and the kind of hardware, it may not be possible to achieve a reasonable framerate while rendering two 1100x1400 views of the world per frame. Hence a lower area of the screen means you have less to render and might mean a better framerate. Rendering two 770x960 frames still means less pixels than a single 1920x1080. That doesn't necessarily make it faster though, since that depends largely on what the bottleneck is in your rendering code. I'm sure older games like Homeworld and the like could probably render dozens of frames at the required resolution in the time it might take something like Arkham Asylum to render a single one.

Bugs

Some people just may not have really understood the distortion shader and may have left out certain components of it, or not properly integrated the code into their own systems.

The work done in the SteamVR API looks pretty intentional, so even though I've marked it as a bug I don't really include it in this category. Rather I've marked it as a bug because there should be a mechanism for SteamVR API clients to specify the FOV they want to support, or perhaps allow the SteamVR API to try to find an appropriate balance between FOV and framerate, though the latter is a much harder engineering problem.

"Pogo" wrote:
I take it you don't know what a lens shader is.

Well you shouldn't.

Jherico posted a nice wall of text describing the problems with all that. Long story short: no.

There's a massive difference between a software renderer like Mental Ray and a real time triangle rasterising gpu based game engine. Standard matrix projected 3d rendering (the way gpus typically work) can't do really wide fov or curved views like you are proposing. It's possible to do it with raytracing/path tracing/etc rendering, but that is vastly slower.
There's ways to simulate it such as rending into faces of a cube map then using a shader to render a wide fov curved view on screen, but they are still the equivalent of what the oculus shaders are doing: render normally then distort.

"Pogo" wrote:
Instead of rendering a normal image and warping it, is it not possible to implement an in-game lens shader that produces the FOV, dimensions and curvature natively? This can be done with software packages like Mental Ray, VRay, Maxwell etc, so I imagine it must be possible within a game engine.

The thing all of the packages you mention have in common is that they're ray tracers (hence the frequent appearance of the word 'ray' in their names).

Ray tracers are fundamentally different from triangle rendering engines like DirectX and OpenGL. Triangle renderers work by sending geometry to a state machine, along with a bunch of information about how the triangles should be handled, such as coloring, lighting, texturing, transparency, etc. The triangles are processed by the rendering API and rasterized into pixels. While much of the state machine and rendering pipeline is programmable, the rasterization portion of it (at least so far) is not. That means you cannot use OpenGL or DirectX to draw curved triangles. At best you can make smaller and smaller triangles to simulate curvature (as well as doing a whole lot of other tricks, usually involving lighting in some form, to make a flat surface appear to be curved). The distortion shader applies a non-linear curvature to the whole scene after it's been rendered.

Triangle renderers work great with modern GPUs because they've sort of evolved together. The renderers do tons of easily decomposible operations that can be divided up among the many parallel processors on the video card. You throw tons of information at the card and it does tons of very small amounts of work very quickly and ends up deciding what triangles go to what pixels on the screen and with what color. In other words, it's working starting with the scene geometry.

Ray tracers on the other hand work in the other direction. Given a pixel on the screen, they compute a virtual ray going from the users eye to that pixel, and compare it against a description of the entire scene to decide what in the scene it hits and where. They the compute additional rays to decide where the ray (or more likely rays since you want to send a ray off at a specific angle to determine what the surface should be reflecting, and then shoot rays off at all the light sources to see which of them is directly lighting this item) should go from there. These virtual rays bounce around the scene until they either hit a fixed limit of bounces or a light source. The end result is that the pixel on the screen gets assigned a color based on all the items that got hit and how they were lit.

Probably the biggest key difference here is the amount of computation done at each step. For triangle renderers, each part of the pipeline only has to know a very small amount of information about the specific triangle they're rendering at the time (or the specific pixel, as affected by a given triangle). Every time the computation for a given ray is done, it has to know, more or less, about every other object in the scene so that it can figure out which of them it intersects with and what sort of action should be taken because of the intersection.

By and large, triangle renderers are used when frame rate is the primary concern. Ray tracers are used when rendering time is not a factor or when the geometry of the rendered objects can't be easily expressed or approximated as triangles. There have been some forays into real-time ray-tracing based applications, but for the most part they're experiments.

The additional computation required to perform the Rift distortion is trivial in terms of ray tracing, since it only affects the very first ray directed for a given pixel on the screen. But at the same time, it's also trivial to do the Rift distortion on a scene that has been rendered by a triangle engine. Given that both approaches are trivial additions to the existing mechanism, and virtually all real-time games and applications are written using triangle engines, why would you bother with the additional constraints of a ray tracer if you didn't have a specific compelling reason to?

"Pogo" wrote:
That's to do with physical screen and lens size. Not the issue.

What I'm talking about is the image rendered to the display, which undercuts that. In many demos, devs only seems to use about 80% of the panel, which undermines the FOV of the Rift.

This image maximizes the real estate of the Rift screen:


This image, like many demos I've seen at native Rift resolution, does not:


The top image yields a much greater sense of immersion when viewed through the Rift, because the image is border to border and my view is limited only by the lenses of the Rift. The bottom image makes me feel like I'm viewing the world through a small pair of tubes because the black borders are distinctly present in my vision. I'm surprised to see few developers taking this into account.

This is spot freakin' on. It's something that has consistently bugged the crap out of me after watching countless Oculus Rift demo videos. Why waste so much horizontal resolution? It's obviously doable as there are the occasional demo or two that utilize the screen like in your second screenshot here.

What's the story? Is this just a matter of beginner developers not understanding best practices or what? This needs to be configurable on the user end if it's not going to be mandatory that all renders maximize the horizontal screen space.

Wrote a blog post on the topic: http://rifty-business.blogspot.com/2014/03/field-of-view-in-oculus-vr.html