Forum Discussion
Why doesn't the rift apply the distortion itself in hardware
Just getting interested in Rift development. The distortion shader required for each component of the stereoscopic image seems to be a function of the screenspace coordinates and the rendered scene fr...
Given that the displacement is a function of where the pixel is within the screen it can be applied to any given pixel at a time regardless of order or sequence in which they are executed. This is the key property of it that allows it to be executed in parallel by the GPU, and would make it possible to apply it while the display scaler (or equivalent module) is receiving the frame instead of having to wait to apply it as a discrete step. It's not rocket science to keep track of which scan line (horizontal coordinate) and derive your vertical position as you process the scan line. The only thing it needs is information on the display configuration (screen size, lens separation distance, and distortion coefficients - all of which are static within a given session and can easily be stored in firmware).
(Edit to clarify: there are two different ways distortion could be performed that I sort of mangled. The way the existing shader works, which is to take the output pixel coordinate and determine the source pixel coordinate from that, or the inverse function which would take a source pixel and transform that into the final output coordinate. Both could work, the difference lies in if you buffer received scanlines until you have everything necessary to generate the distorted scanline, or if you distort them and then buffer the output until your outgoing scanline is complete. I was assuming the former, but the latter is equally theoretically valid. I won't even attempt to compare the pros/cons of the two though.)
Keep in mind that in order for the scaler in a display to upsample or downsample a frame it is operating spatially both within the scan line and across multiple scan lines too. The key difference here is that a distortion stage would require more buffering because the radial distance is significantly greater.
The key to it all would be proper timing. Your typical latency would be (off the top of my head) a function of the maximum vertical shift in scan lines that you incur by doing the distortion - which is going to be some fraction of the time it takes to transmit the entire frame. If you try to push the distorted frame sooner than that you're going to stall, which is bad. But if you push after that, you get an effect not to dissimilar to pipelining inside CPUs - you've greatly reduced the idle time by not having to wait to receive the entire frame to perform a discrete step.
It would definitely require an ASIC to implement, but the benefit of it is that because it wouldn't have to be designed around traditional GPU workloads you could optimize it to apply filtering that is traditionally inefficient on the GPU - e.g., Lanczos or Bicubic filtering w/ Chromatic Aberration Correction - or even piecewise functions that are capable of applying different filtering at different radial lengths.
While it would definitely drive cost up (e.g., the Gsync module, which you could probably use to implement a proof of concept, runs something like $300 on it's own. that's less than ideal, but it's high cost is in large part due to it's use of an FPGA.), that would come down with scale.
As far as complexity... which is better - internalizing the cost/complexity of applying distortion shading (which is both mandatory for an image to be viewed correctly and can impact motion sickness if done correctly) or externalizing the cost across an entire industry? History typically shows that the former wins out in almost any given market because it allows standardization and greater economies of scale, whereas the former typically leads to a lot more variation in experience and a lot of long term problems with fragmentation (in this case, different developers doing different things, along with serious compatibility concerns if software has to be updated to support different distortion techniques for newer HMDs).
I'm not suggesting this is something that should be implemented on DK2 or the consumer Rift. I'm just pointing out that it is significantly more feasible than you are suggesting and would offer some long term platform benefits (e.g., easier transition to new optics, etc).
(Edit to clarify: there are two different ways distortion could be performed that I sort of mangled. The way the existing shader works, which is to take the output pixel coordinate and determine the source pixel coordinate from that, or the inverse function which would take a source pixel and transform that into the final output coordinate. Both could work, the difference lies in if you buffer received scanlines until you have everything necessary to generate the distorted scanline, or if you distort them and then buffer the output until your outgoing scanline is complete. I was assuming the former, but the latter is equally theoretically valid. I won't even attempt to compare the pros/cons of the two though.)
Keep in mind that in order for the scaler in a display to upsample or downsample a frame it is operating spatially both within the scan line and across multiple scan lines too. The key difference here is that a distortion stage would require more buffering because the radial distance is significantly greater.
The key to it all would be proper timing. Your typical latency would be (off the top of my head) a function of the maximum vertical shift in scan lines that you incur by doing the distortion - which is going to be some fraction of the time it takes to transmit the entire frame. If you try to push the distorted frame sooner than that you're going to stall, which is bad. But if you push after that, you get an effect not to dissimilar to pipelining inside CPUs - you've greatly reduced the idle time by not having to wait to receive the entire frame to perform a discrete step.
It would definitely require an ASIC to implement, but the benefit of it is that because it wouldn't have to be designed around traditional GPU workloads you could optimize it to apply filtering that is traditionally inefficient on the GPU - e.g., Lanczos or Bicubic filtering w/ Chromatic Aberration Correction - or even piecewise functions that are capable of applying different filtering at different radial lengths.
While it would definitely drive cost up (e.g., the Gsync module, which you could probably use to implement a proof of concept, runs something like $300 on it's own. that's less than ideal, but it's high cost is in large part due to it's use of an FPGA.), that would come down with scale.
As far as complexity... which is better - internalizing the cost/complexity of applying distortion shading (which is both mandatory for an image to be viewed correctly and can impact motion sickness if done correctly) or externalizing the cost across an entire industry? History typically shows that the former wins out in almost any given market because it allows standardization and greater economies of scale, whereas the former typically leads to a lot more variation in experience and a lot of long term problems with fragmentation (in this case, different developers doing different things, along with serious compatibility concerns if software has to be updated to support different distortion techniques for newer HMDs).
I'm not suggesting this is something that should be implemented on DK2 or the consumer Rift. I'm just pointing out that it is significantly more feasible than you are suggesting and would offer some long term platform benefits (e.g., easier transition to new optics, etc).