Why doesn't the rift apply the distortion itself in hardware

Adding a video digitizer and a GPU to the Rift would add significantly to its cost, and increase the time to market. It could probably be done with a custom ASIC, but that would take time to design, test, and manufacture in the quantity needed at a cost within required budget constraints, with the power consumption and bandwidth requirements that will work with an HD consumer Rift.

You can buy inexpensive boxes that convert HDMI to synthetic SBS 3D, so such a thing is possible, and may even be doable with existing ASIC components, but would need custom firmware at least, and may add significant latency in the process.

It would probably delay time to market, if nothing else. At least that is how I see possible complications that may need to be taken into account to do as you suggest...

If they were selling millions of the things, it'd probably make sense; the type of ASIC we're talking about is incredibly simple compared to even a mobile GPU, so you'd end up with big upfront costs but very little manufacturing costs. As such, it probably doesn't make sense in the relatively small volumes we'd see for the V1 (they used off-the-shelf chips for their custom sensor board, for example).

Then again, I can definitely see benefits to Oculus doing their own custom display controllers if they start shipping volumes where that makes economic sense, and that'd be the perfect place to include such a thing. And this is all assuming that there isn't already image processing chips on the market that could do the warping in an affordable manner.

"kierenj" wrote:
So why wouldn't the rift offload this processing - allow 'normal' rendering in splitscreen fashion, but then apply the distortion in hardware, the other end of the HDMI link?

If you're suggesting that the Rift be able to render pre-existing standard SBS stereo content like this, then bear in mind that there's supposed to be a 1:1 correlation between the screenspace coordinate and the angle of view. At best you'd have to scale the content down so that all of the image would be visible inside field of view, without pushing the corners too far out, basically mimicking the 3D moviegoing experience. Allowing the Rift to handle 'conventional' stereo content would require compromises that render useless a number of valuable features of the Rift, such as the wide field of view, as well as increasing the chance of motion sickness due to a mismatch between the game FOV and the perceived FOV.

If you're suggesting that this would allow games to work with less effort, then you must remember that there are other elements to the 3D effect that are rift specific, such as doing the projection matrix offset, that are also required for proper rendering, so this doesn't really drastically reduce the amount of work required to get something working with the Rift (although it would allow toolkits that don't support post processing, such as Unity Free, to work). Finally, the distortion shader, while somewhat complex, isn't really an expensive operation on modern GPU hardware, so given the increased cost in hardware to support it that Geekmaster mentioned, what really is the advantage to doing this?

"geekmaster" wrote:
Adding a video digitizer and a GPU to the Rift would add significantly to its cost, and increase the time to market. It could probably be done with a custom ASIC, but that would take time to design, test, and manufacture in the quantity needed at a cost within required budget constraints, with the power consumption and bandwidth requirements that will work with an HD consumer Rift.

You can buy inexpensive boxes that convert HDMI to synthetic SBS 3D, so such a thing is possible, and may even be doable with existing ASIC components, but would need custom firmware at least, and may add significant latency in the process.

It would probably delay time to market, if nothing else. At least that is how I see possible complications that may need to be taken into account to do as you suggest...

Why would you need a digitizer and GPU? Given that the distortion function can be reduced to a fixed function for any given HMD, you don't actually need to apply this as a discrete step using a GPU. Instead, you could easily take the route that Gsync took and replace the dedicated scaler with one that is capable of applying a distortion as it receives scan lines.

It would still add cost and require additional time to market, though, since you're losing the advantage of using a high-volume produced scalar.

On the plus side, it would mean having a much more consistent distortion function across applications and would allow changes in display configurations with only minimal developer intervention (you would still have to account for FOV differences, but eh).

Edit: just to be clear, the above is a GROSS oversimplification. Point I'm getting at is that you don't need to apply the distortion as a discrete step if you have control over the hardware in the HMD.

"dghost" wrote:
Why would you need a digitizer and GPU? Given that the distortion function can be reduced to a fixed function for any given HMD, you don't actually need to apply this as a discrete step using a GPU. Instead, you could easily take the route that Gsync took and replace the dedicated scaler with one that is capable of applying a distortion as it receives scan lines. ...

It is RADIAL distortion, with vertical displacement in addition to horizontal, varying with radial distance from lens centers. You cannot do that per scanline as you suggested. Depending on how it is done, it could easily add an entire frame delay to the latency.

Okay, with HDMI input you would probably not need a digitizer, although you would need linear interpolation, or accept full-pixel displacement errors. Anything that modifies the graphic content of the framebuffer is essentially a GPU, although perhaps a simple GPU, which could be implemented with an ASIC, as I mentioned previously. Adding interpolation into the mix requires even more GPU-like hardware.

And that adds to the cost and complexity, including addititonal firmware support. And moving configuration settings from CPU drivers to firmware or hardware would also add to complexity.

Given that the displacement is a function of where the pixel is within the screen it can be applied to any given pixel at a time regardless of order or sequence in which they are executed. This is the key property of it that allows it to be executed in parallel by the GPU, and would make it possible to apply it while the display scaler (or equivalent module) is receiving the frame instead of having to wait to apply it as a discrete step. It's not rocket science to keep track of which scan line (horizontal coordinate) and derive your vertical position as you process the scan line. The only thing it needs is information on the display configuration (screen size, lens separation distance, and distortion coefficients - all of which are static within a given session and can easily be stored in firmware).

(Edit to clarify: there are two different ways distortion could be performed that I sort of mangled. The way the existing shader works, which is to take the output pixel coordinate and determine the source pixel coordinate from that, or the inverse function which would take a source pixel and transform that into the final output coordinate. Both could work, the difference lies in if you buffer received scanlines until you have everything necessary to generate the distorted scanline, or if you distort them and then buffer the output until your outgoing scanline is complete. I was assuming the former, but the latter is equally theoretically valid. I won't even attempt to compare the pros/cons of the two though.)

Keep in mind that in order for the scaler in a display to upsample or downsample a frame it is operating spatially both within the scan line and across multiple scan lines too. The key difference here is that a distortion stage would require more buffering because the radial distance is significantly greater.

The key to it all would be proper timing. Your typical latency would be (off the top of my head) a function of the maximum vertical shift in scan lines that you incur by doing the distortion - which is going to be some fraction of the time it takes to transmit the entire frame. If you try to push the distorted frame sooner than that you're going to stall, which is bad. But if you push after that, you get an effect not to dissimilar to pipelining inside CPUs - you've greatly reduced the idle time by not having to wait to receive the entire frame to perform a discrete step.

It would definitely require an ASIC to implement, but the benefit of it is that because it wouldn't have to be designed around traditional GPU workloads you could optimize it to apply filtering that is traditionally inefficient on the GPU - e.g., Lanczos or Bicubic filtering w/ Chromatic Aberration Correction - or even piecewise functions that are capable of applying different filtering at different radial lengths.

While it would definitely drive cost up (e.g., the Gsync module, which you could probably use to implement a proof of concept, runs something like $300 on it's own. that's less than ideal, but it's high cost is in large part due to it's use of an FPGA.), that would come down with scale.

As far as complexity... which is better - internalizing the cost/complexity of applying distortion shading (which is both mandatory for an image to be viewed correctly and can impact motion sickness if done correctly) or externalizing the cost across an entire industry? History typically shows that the former wins out in almost any given market because it allows standardization and greater economies of scale, whereas the former typically leads to a lot more variation in experience and a lot of long term problems with fragmentation (in this case, different developers doing different things, along with serious compatibility concerns if software has to be updated to support different distortion techniques for newer HMDs).

I'm not suggesting this is something that should be implemented on DK2 or the consumer Rift. I'm just pointing out that it is significantly more feasible than you are suggesting and would offer some long term platform benefits (e.g., easier transition to new optics, etc).

You could do the distortion in hardware, but you don't want to:

* You can sometimes add the distortion to another post-process (e.g. simple color grading, tonemapping) while distortion in HW requires you to store large parts of the frame in HW before you start to distort -> you add latency.
* You want to render offscreen at a higher resolution to reduce filter artifacts -> your undistorted image might have a higher resolution than your HDMI/DVI would be able to send at 60Hz (think of higher resolution Rifts)
* You have more control over the quality of filtering.
* If your application does not use rasterisation but ray-tracing, you can distort during rendering (also with rasterisation if the geometry gets tessellated enough)

The "benefit" of 1-2 msec less rendering time (worst case!) is not worth the loss in options, flexibility and added latency (probably > 2msec, 17msec worst case (buffer one frame at 60Hz))

Even those small gains are not realistic. Somewhere, at some point, a fullscreen render pass needs to happen. Whether it's done in specialized hardware or on your GPU, it will take time and add latency.

In the end your overall latency would probably get worse, because your GPU is already very good at this kind of processing. Designing something that can do it faster would be challenging and expensive, and would be outperformed in the next GPU generation.

If you are concerned about performance, you should use a pre-generated lookup texture by procedurally encoding the distorted coordinates.

And now you could upload nice screen casts to Youtube everyone will clearly recognize as Rifts (and watch them with Rift)!
:D
And there is a dark place between circles to insert author image from the outer camera ;)

ps. also you need to do antialiasing and combine overlapping pixels for outer regions in HW.
pps. and you need not render peripheral at the same detail as central part (for ray-tracing).