Bicubic Warp Reconstruction

I've just tried this and I do think your version looks a little better on the edges of things like barrels, window sills, etc. It does look a little 'softer' somehow, but it also doesn't have as much of the 'random illogical' colors on slanted lines (bright greens and the like on a dark metal barrel rim for instance, and which 'shimmer' as you move and your view changes slightly).

I hadn't even really noticed those aliasing effects before, but switching between the two, I can see that your version has less of them. I'd say that overall your version looks better.

Also, I couldn't get the EXE to run until I copied msvcr100d.dll from another demo into the same folder. The stock demo runs fine.

Thanks for this.

Oh, and as far as frame rate, it ran as smoothly as the stock demo on my GTX670 at 1920x1080.

"jherico" wrote:

"TeraBit" wrote:
Hmmm what am I doing? :? Heh, still seems to work though.. 8-) Theories, you gotta love em.

Sounds like it might just be a manual implementation of multi-sampling.

Sorta-but-not-quite. This seems to actually be more like implementing a variable blur, which actually would improve image quality over the native, non-antialiased image. Antialiasing would achieve a similar effect, albeit producing more accurate results due to being able to more precisely represent the scene.

Improving post-distortion image quality is surprisingly difficult to do as it actually introduces aliasing/artifacts of it's own. This is part of the reason why simply rendering at a higher resolution and then downsampling the final image actually appears to improve quality so much - not only are you effectively applying SSAA to the scene, but it also smooths out some of the irregularities that get introduced by the distortion itself.

And @TeraBit - sadly, my Rift is on loan at the moment so I can't check it out for a few more days. I'll probably drop it into my test bench to see what it says, but I won't get a chance to do subjective tests for a little while.

Thanks for the feedback. :)

I agree that super sampling is a much better way of getting the correct result. It would be interesting to see what a little of both would do. It's hard to put more information in than is actually there. 8-)

I have not tested it, but here is a shortened version of your code (Triangular and BiCubic):

float Triangular(float f)
{
   return 1.0-abs(f*0.5);
}

float4 BiCubic(Texture2D textureSampler, float2 TexCoord, float TexelScale, SamplerState LNR)
{
   float2 size = float2(1280, 800) * TexelScale;
   float2 texelSize = 1.0/size;
   float2 ab = frac(TexCoord*size);
   float2 TexCoord1 = (floor(TexCoord*size)+0.5)/size;
	
   float4 nSum = float4(0);
   float4 nDenom = float4(0);
   
   for( int m = -1; m <=2; m++ )
   {
      for( int n =-1; n<= 2; n++)
      {
         float2 TC = TexCoord1 + texelSize * float2(m, n);
         float4 vecData = textureSampler.SampleLevel(LNR, TC, 0);
         float4 coeff = float4( Triangular(float(m)-ab.x) * Triangular(float(n)-ab.y) );
         nSum += vecData * coeff;
         nDenom += coeff;
      }
   }
   float4 retval = nSum / nDenom;
   return clamp(retval, 0.0, 1.0);
}

But besides that I'm wondering, if this is actually Bicubic reconstruction, because you are using a triangle-shaped kernel. But the kernel needed for bicubic reconstruction should look rather like this:

Thanks for the updated code! 8-)

The Triangular function is a lot more sensible now. :)

The 'Bicubic' one didn't initially work, so I had to amend it to this:

	float4 BiCubic(Texture2D textureSampler, float2 TexCoord, float TexelScale, SamplerState LNR)
{
   float2 size = float2(1280, 800) * TexelScale;
   float2 texelSize = float2(1.0/size);
   float2 ab = frac(TexCoord*size);
   float2 TexCoord1 = (floor(TexCoord*size)+0.5)/size;
   
   float4 nSum = float4(0.0,0.0,0.0,0.0);
   float4 nDenom = float4(0.0,0.0,0.0,0.0);
   
   for( int m = -1; m <=2; m++ )
   {
      for( int n =-1; n<= 2; n++)
      {
         float2 TC = TexCoord1 + texelSize * float2(m, n);
         float4 vecData = textureSampler.SampleLevel(LNR, TC, 0);
		  float f = Triangular(float(m)-ab.x);
		  float f1 = Triangular(float(n)-ab.y);
         float4 coeff = float4(f,f,f,f) * float4(f1,f1,f1,f1);
         nSum += vecData * coeff;
         nDenom += coeff;
      }
   }
   float4 retval = nSum / nDenom;
   return clamp(retval, 0.0, 1.0);
}

That said, both together added about 4 FPS on my system overall, so worthwhile I'd say. :)

But besides that I'm wondering, if this is actually Bicubic reconstruction, because you are using a triangle-shaped kernel.

The name of the function kind of gives it away. It appears to be a fast rough-and-ready approximation of the spline needed (it should be noted that I did not write the Triangular function, or most of the Bicubic one, but simply converted them from GLSL examples I found floating around).

That said, they seem to do what I wanted them to do, so for the purposes of my demo it does the job. If you know of a simple drop in replacement function for Triangular, I'm happy to give it a go. ;)

"TeraBit" wrote:
The 'Bicubic' one didn't initially work, so I had to amend it to this:

I knew from GLSL that you can do something like this:

vec4(0)

But it seems this is not possible with HLSL, which I never used.

One small optimization for your code:
I would replace that:

float f = Triangular(float(m)-ab.x);
float f1 = Triangular(float(n)-ab.y);
float4 coeff = float4(f,f,f,f) * float4(f1,f1,f1,f1);

with this:

float f = Triangular(float(m)-ab.x) * Triangular(float(n)-ab.y);
float4 coeff = float4(f,f,f,f);

Three multiplications less.

The formula for the bicubic kernel is a bit complicated. I generally prefer the lanczos-filter, which is similar. A drop-in replacement for Triangular:

float Sinc(float x) {
   float pi_x = x*3.14159;
   return sin(pi_x)/pi_x;
}
float Lanczos(float x) {
   if (abs(x) < 0.001) return 1.0;
   else return Sinc(x)*Sinc(x/2);
}

You then have to use Lanczos instead of Triangular. It has the same kernel-size as Triangular, so you don't have to change the boundaries of your two nested for-loops.
I think your framerate will decrease, because the sine-function might be pretty expensive to compute. A solution for this would be using another texture as lookup-table containing the pre-computed Lanczos-kernel.
By the way, here is a comparison of the Lanczos-kernel and your Triangular one: 8-)
http://www.wolframalpha.com/input/?i=sinc%28x*pi%29*sinc%28x*pi%2F2%29%2C+1-abs%28x%2F2%29+from+-2+to+2

Thanks for the updated code. I've integrated it and will post another demo once I've got it working as expected.

So far the results are a little more subtle than the Triangular Filter. So need to find the right balance of settings. 8-)

I've been waiting for someone to take the time to try this. There may be more implementation details to work out (and perhaps other reconstruction methods to try) but it is definitely the correct approach to get optimal image quality out of the low res display, and really the only way to avoid shimmering without loss.

Traditional antialiasing is just a special case of what is being done here, made simpler by the lack of distortion. And for those worried about loss of detail, this actually brings in more detail that would otherwise be lost due to undersampling, so more rendered pixels contribute to the final image.

This does not eliminate the need for supersampling though, due to the magnification of the hmd warp. It might greatly reduce or eliminate the need for MSAA though, making it useful for deferred rendering and other post processing techniques that DX9 can't handle very well with supersampling.

Well, I never got around to posting an updated demo. :oops:

I tried the Lanczos filter as opposed to the Triangular filter. It seems that the Lanczos one is much more weighted toward retaining the detail of each pixel than blending them as the Triangular one is. As a result the softening of the edges is a little too subtle to notice. I stood there toggling back and forth between the Lanczos and Standard one, and couldn't see see a world of difference between them. It seems the triangular one is better for this resolution.

I suspect that if the render buffer was bigger, it would produce better results.

This also made me think that it might be interesting to apply a contrast based edge detection filter on it, so that it only applies the blur above a certain contrast threshold (I used a similar function on the CUDA Raytracing demo to do some AA on high contrast items). Truth be told I've been very busy with TeraIDE, so not had much time to tinker. Would be easier to fiddle with this stuff if I had access to the warp shader code in the engines (UDK / Unity). :roll:

"TeraBit" wrote:
I tried the Lanczos filter as opposed to the Triangular filter. It seems that the Lanczos one is much more weighted toward retaining the detail of each pixel than blending them as the Triangular one is. As a result the softening of the edges is a little too subtle to notice. I stood there toggling back and forth between the Lanczos and Standard one, and couldn't see see a world of difference between them. It seems the triangular one is better for this resolution.

Yes, the main-lope of the Lanczos-filter is one-half of that from the Triangular one, so it smoothes much less. If you increase the filter-size you can compensate for that. Please try this code:

#define FILTER_SIZE 2

float Lanczos(float x) {
	if (abs(x) < 0.001) return 1.0;
	x *= 2.0/FILTER_SIZE;
	else return Sinc(x)*Sinc(x/2);
}
float4 BiCubic(...) {
	...
	for( int m = -FILTER_SIZE+1; m <= FILTER_SIZE; m++ ) {
		for( int n = -FILTER_SIZE+1; n <= FILTER_SIZE; n++) {
			...
		}
	}
	...
}

If FILTER_SIZE is set to 2, you will get the same result as before. But if you increase it (for instance 3, 4 or 5) you will get smoother results, but the performance will decrease significantly.

Another idea is to use a different filter with just one main lope, for example the cosine-filter (which has in this case the same size as your triangular):

float CosFilter(float x) {
	return cos(x*(3.14159*0.25));
}

"TeraBit" wrote:
This also made me think that it might be interesting to apply a contrast based edge detection filter on it, so that it only applies the blur above a certain contrast threshold (I used a similar function on the CUDA Raytracing demo to do some AA on high contrast items).

I fear that all the extra-steps needed for finding the high-contrast areas will cancel out the performance-gain of filtering less pixels, because finding high-contrast areas respectively edge detection basically is just another filtering-process applied to all pixels. For raytracing this can be a good idea, because raytracing usually is much more computationally intensive than linear filtering with a small kernel.