AA vs downscaling from higher resolutions

So here is the basic idea I am working on right now. The supersampling looks really good. Everyone is in agreement there. It does however have a negative effect on frame rate. It can really tax the fill rate of your gfx card. And while looking good is important, in VR land latency is king. So what I want to do is to make things look as good as possible while keeping a good framerate (duh, right?)

Well it's really difficult to get that balance dialed in just right. It's pretty difficult to get it right on my machine, let alone creating some settings that will apply across the board to users running a myriad of hardware. So what I want to create is a dynamic, reactive quality adjustment system. With a system like this, if the user chooses too high quality (they always will, because everyone wants teh highest qualities) it will automatically start scaling down to a reasonable level to maintain a good latency.

One method that might be used in this implementation is a velocity-dependent-buffer-swap. The theory is this: a slower-than-optimal frame rate (~40-50fps) isn't all that bad if you aren't moving your head around too much. When you start moving your head around is when you feel the effects of the latency. Also, when you are moving your head quickly, you cannot see the scene as well (with the DK the whole world gets smeared, but the human visual system doesn't pick up on fine details during fast movement anyways). So what I'm doing is this: keep the high quality rendering if the user isn't moving her head around too much, but if she moves her head quickly, swap to a lower quality render buffer to keep latency low during the fast movement.

Here is an example. If I use a 6x render buffer my fps drops to like 40 in some scenes. This works alright if I'm holding my head pretty still, but if I move my head quickly at all it becomes really choppy and nauseating. However, this implementation will realize that and swap to a lower quality buffer when you start moving your head quickly. So it swaps to a 2x buffer and you get 60 fps while moving your head quickly. This helps prevent sickness while allowing users to over-extend their systems with high quality settings.

This is just one of the techniques that will need to go into sickness-preventing, quality-optimizing rendering. This is more of a knee-jerk reaction kind of thing. It will help keep things running smoothly in a crunch, while a better solution is being determined. Meanwhile, a more long-term performance analyzer will be monitoring the rendering over longer periods. If you are averaging like 40 fps, it will start making changes to decrease latency.

Any suggestions on optimization techniques?

Hey boone188, I'm really loving this bicubic shader... thanks for hooking us up! I've noticed that some of my mouse scripts (like doing a simple mouse pick) no longer work for some reason. Any idea what could be causing this?

Pretty great idea, Boone -- to dynamically adjust the supersampling size based on head movement. That'd be really interesting to see working as a proof of concept. I wonder how you would determine when the head is really starting to move though? Comparing if the head rotation's angle since the last frame is > X amount?

Super sampling does always look good, but as an antialiasing technique it is not good from a quality/performance perspective.

Some supersampling is absolutely necessary for the Rift due to how the distortion works. You should use the OVR SDK distortion scale to determine how much. Without it you are rendering the rough equivalent of a 480p image on a 720p display. Given so few pixels to work with, that is a crime.

However, beyond that distortion scale you are just doing inefficient AA. High MSAA will give you pretty much the same result at a lower cost, and it won't shimmer nearly as much.

The shimmering is due to undersampling, you are completely skipping over a lot of texels when you do the distortion, so there is a lot of randomness (and outright waste) in which texels actually end up on the screen, which is unavoidable if you do not do some downsample filtering of the high res image. MSAA does this filtering for you when it resolves the subsamples into a final image, though in this case it resolves pre-distortion which does lead to some undersampling but it does not increase with MSAA level and is bounded by the distortion scale.

This is very important for VR because shimmering will differ between eyes creating extra stereo disparities that your brain can't easily deal with. But the ideal would be to run a proper downsampling image reconstruction filter, which would smooth out the sampling errors caused by the distortion. Nobody seems to have tackled that yet.

Incidentally the supersampling code in the SDK does not seem to work at all anymore. I just get a blue screen.

Oh and I have tried dynamic resolution rendering in Unity in the past as a way to smooth out performance, but changing viewports seems to allocate a bunch of memory in a lot of cases, especially when using render textures and terrain for some reason. With deferred lighting it actually creates new render targets at the viewport size, which is ruinously wasteful. Not really something you can adjust per-frame.

"owenwp" wrote:
Oh and I have tried dynamic resolution rendering in Unity in the past as a way to smooth out performance, but changing viewports seems to allocate a bunch of memory in a lot of cases, especially when using render textures and terrain for some reason. With deferred lighting it actually creates new render targets at the viewport size, which is ruinously wasteful. Not really something you can adjust per-frame.

I've got it working pretty well right now. The switch doesn't cause too much of a hit, and definitely gives better results than sticking with the slow rendering. I can get around 40 fps running at 8x but it shoots up to 60 fps when turning my head quickly. The tricky part is determining when to make the switch.

Interesting idea boone, I'd love to see this in action and test it out!

One thing however, 8x sounds a little crazy to me! That's 5120x6400!! In my tests (if you're rendering the distortion with bicubic filtering), there's no quality increase above 3x which makes sense.

"owenwp" wrote:
Some supersampling is absolutely necessary for the Rift due to how the distortion works. You should use the OVR SDK distortion scale to determine how much.

In this thread, it seems to indicate that a scaling factor of 1.71 is optimal;
viewtopic.php?f=20&t=3789#p48286

Perhaps the Unity integration should use this by default?

"Teddy0k" wrote:
Interesting idea boone, I'd love to see this in action and test it out!

One thing however, 8x sounds a little crazy to me! That's 5120x6400!! In my tests (if you're rendering the distortion with bicubic filtering), there's no quality increase above 3x which makes sense.

"owenwp" wrote:
Some supersampling is absolutely necessary for the Rift due to how the distortion works. You should use the OVR SDK distortion scale to determine how much.

In this thread, it seems to indicate that a scaling factor of 1.71 is optimal;
viewtopic.php?f=20&t=3789#p48286

Perhaps the Unity integration should use this by default?

8X is crazy. I only went that high to cause slow-down. I needed to emulate the effects of someone running 2-4X on a slower machine. I am able to see quality increase up to around 4X.

"Teddy0k" wrote:
Interesting idea boone, I'd love to see this in action and test it out!

One thing however, 8x sounds a little crazy to me! That's 5120x6400!! In my tests (if you're rendering the distortion with bicubic filtering), there's no quality increase above 3x which makes sense.

"owenwp" wrote:
Some supersampling is absolutely necessary for the Rift due to how the distortion works. You should use the OVR SDK distortion scale to determine how much.

In this thread, it seems to indicate that a scaling factor of 1.71 is optimal;
viewtopic.php?f=20&t=3789#p48286

Perhaps the Unity integration should use this by default?

The optimal scaling of back buffer depends on how much scaling the distortion shader is applying. For DK1, if you stretch to the outer edges it's 1.71. If you are stretching to the inner edges it's something like 1.41 (I'd have to look it up when I get home). It's important to note that this isn't super scaling at all - scaling the back buffer by the same amount as the distortion shader is necessary if you want to ensure that the final image retains the same pixel density and quality as if the distortion was unscaled.

Applying "pseudo" 4X SSAA can then be done by then doubling the resolution of the back buffer and making sure you enable bilinear filtering. For most cases this will result in having a back buffer that's ~2.8x-3.4x the resolution of the front buffer. Scaling the buffer past that will create artifacting because bilinear filtering only samples a 2x2 grid of pixels when calculating a final RGB value - the past (front * distortion scale * 2) will mean that you now have gaps where pixels aren't being sampled and represented in the front buffer. You could mitigate that by implementing a custom bilinear filtering kernel that samples (n x n) pixels where n^2 is the AA level, but you're going to really hurt for performance. You could probably do 16x without too much of a performance hit using 4 bilinear samples, but you're really getting into the "you've made some bad life choices" territory at that point.

Bicubic filtering actually can produce really some nice results with 4xSSAA, but generally the kind of AA you get out of this are not going to compare to the quality you'll get out of applying MSAA/FXAA to an optimally scaled back buffer. Plus, it causes a massive performance hit that is going to hurt if you want it to work with a HD DK.

Edit: fixed some errors from being tired and not focused on graphics work. I should also add that because the distortion is non-linear in nature, simply scaling the back buffer isn't going to give you the same results as an actual FSAA implementation would. You'd need to render everything, distortion included, at a higher resolution and then downsample the result to achieve that.

I see now that DirectX 9 is basically incapable of doing the optimal method of supersampling at distortion scale with MSAA outlined above.

The typical DX9 render to texture trick is to draw into the backbuffer using a viewport that is the same size as your render texture, which lets you use MSAA, then copy that rect from the backbuffer into the texture (already wasteful). But of course if your render target is bigger than the backbuffer, this won't work at all. Not something that would likely come up in the days before VR, but now it is a serious problem.

This is a very good reason for everyone here to make the jump to DirectX 11, because DirectX 9 will never give optimal image quality on the rift without using wasteful and outlandish render pipeline tricks. I don't think Unity takes advantage of the DX11 RTT MSAA API yet though.