Bicubic Warp Reconstruction

TeraBit, can you show us the relevant shader-code, which performs the bicubic reconstruction?

"mrjazz" wrote:
TeraBit, can you show us the relevant shader-code, which performs the bicubic reconstruction?

Sure, I was planning to do that, but it's messy and I wanted to comment a couple of bits first.

Also, it's just a proof of concept really rather than a serious implementation. I'm not even sure I'm calculating some of the elements properly.

I'll post what I have soon (I need to copy it out of the C++ code I hacked it into)...

Disclaimer. This is my first HLSL shader, so I may be doing what may appear to be some very strange things :lol:

Texture2D Texture : register(t0);
SamplerState Linear;
float2 LensCenter;
float2 ScreenCenter;
float2 Scale;
float2 ScaleIn;
float4 HmdWarpParam;
float4 ChromAbParam;
float MaxSharpness;
float MinSharpness;
float WarpMultiplier;
float ShowGradient;


float2 HmdWarp(float2 in01)
{
   float2 theta = (in01 - LensCenter) * ScaleIn; // Scales to [-1, 1]
   float  rSq = theta.x * theta.x + theta.y * theta.y;
   float2 theta1 = theta * (HmdWarpParam.x + HmdWarpParam.y * rSq + HmdWarpParam.z * rSq * rSq + HmdWarpParam.w * rSq * rSq * rSq);
   return LensCenter + Scale * theta1;
}

float Triangular(float f)
{
   f = f / 2.0;
   if( f < 0.0 )
   {
	return ( f + 1.0 );
   }
   else
   {
	return ( 1.0 - f );
   }
   return 0.0;
}

float4 BiCubic(Texture2D textureSampler, float2 TexCoord, float TexelScale, SamplerState LNR )
{
   float fWidth = 1280.0 * TexelScale;
   float fHeight = 800.0 * TexelScale;
   float texelSizeX = 1.0 / fWidth; //size of one texel 
   float texelSizeY = 1.0 / fHeight; //size of one texel 
   float4 nSum = float4( 0.0, 0.0, 0.0, 0.0 );
   float4 nDenom = float4( 0.0, 0.0, 0.0, 0.0 );
   float a = frac( TexCoord.x * fWidth ); // get the decimal part
   float b = frac( TexCoord.y * fHeight ); // get the decimal part
	
   int nX = int(TexCoord.x * fWidth);
   int nY = int(TexCoord.y * fHeight);
   float2 TexCoord1 = float2( float(nX) / fWidth + 0.5 / fWidth, float(nY) / fHeight + 0.5 / fHeight );
	
   for( int m = -1; m <=2; m++ )
      {
         for( int n =-1; n<= 2; n++)
         {
            float2 TC = TexCoord1 + float2(texelSizeX * float(m), texelSizeY * float(n));
            float4 vecData = textureSampler.SampleLevel(LNR, TC,0);
            float f  = Triangular( float( m ) - a );
            float4 vecCooef1 = float4( f,f,f,f );
            float f1 = Triangular( -( float( n ) - b ) );
            float4 vecCoeef2 = float4( f1, f1, f1, f1 );
            nSum = nSum + ( vecData * vecCoeef2 * vecCooef1  );
            nDenom = nDenom + (( vecCoeef2 * vecCooef1 ));
         }
      }  
      float4 retval = float4(nSum / nDenom);
      return clamp(retval,0.0,1.0);
}

float4 main(in float4 oPosition : SV_Position, in float4 oColor : COLOR, in float2 oTexCoord : TEXCOORD0) : SV_Target
{

   float fWidth = 1280.0;
   float fHeight = 800.0;
   float texelSizeX = 1.0 / fWidth; //size of one texel 
   float texelSizeY = 1.0 / fHeight; //size of one texel 
	 
   float2 LFT = HmdWarp(oTexCoord + float2(-texelSizeX, 0));
   float2 RIT = HmdWarp(oTexCoord + float2(texelSizeX, 0));

   float2 UPT = HmdWarp(oTexCoord + float2(0, -texelSizeY));
   float2 DWT = HmdWarp(oTexCoord + float2(0, texelSizeY));

   float XDIS = abs(LFT.x - RIT.x);
   float YDIS = abs(UPT.y - DWT.y);

   float texsize = clamp(MaxSharpness - (((XDIS / texelSizeX) + (YDIS / texelSizeY)))*WarpMultiplier,MinSharpness,MaxSharpness);

   float2 theta = (oTexCoord - LensCenter) * ScaleIn; // Scales to [-1, 1]
   float  rSq = theta.x * theta.x + theta.y * theta.y;
   float2 theta1 = theta * (HmdWarpParam.x + HmdWarpParam.y * rSq + HmdWarpParam.z * rSq * rSq + HmdWarpParam.w * rSq * rSq * rSq);
   
   // Detect whether blue texture coordinates are out of range since these will scaled out the furthest.
   float2 thetaBlue = theta1 * (ChromAbParam.z + ChromAbParam.w * rSq);
   float2 tcBlue = LensCenter + Scale * thetaBlue;
   if (any(clamp(tcBlue, ScreenCenter-float2(0.25,0.5), ScreenCenter+float2(0.25, 0.5)) - tcBlue)) return 0;
   
	
   // Now do blue texture lookup.
   float  blue = BiCubic(Texture, tcBlue, texsize, Linear).b; //Texture.Sample(Linear, tcBlue).b;
   
   // Do green lookup (no scaling).
   float2 tcGreen = LensCenter + Scale * theta1;
   float4 greenColor = BiCubic(Texture, tcGreen, texsize, Linear);
   float  green =  greenColor.g;
   float  alpha = greenColor.a;
   
   // Do red scale and lookup.
   float2 thetaRed = theta1 * (ChromAbParam.x + ChromAbParam.y * rSq);
   float2 tcRed = LensCenter + Scale * thetaRed;
   float  red = BiCubic(Texture, tcRed, texsize, Linear).r;
   if(ShowGradient != 0.0f)
   {
      red   = (texsize / MaxSharpness)-( MinSharpness / MaxSharpness);
      green = (texsize / MaxSharpness)-( MinSharpness / MaxSharpness);
      blue  = (texsize / MaxSharpness)-( MinSharpness / MaxSharpness);
      alpha = 1.0;
   };
	   
   return float4(clamp(red,0.0,1.0),clamp(green,0.0,1.0),clamp(blue,0.0,1.0),alpha);
	 
};

Interesting. I've been testing similar techniques recently and was having difficulty getting any quantifiable improvements out of Bicubic filtering. That said, I was testing an optimized 4-sample implementation, and my next step was to move to the full 16-sample version. I have standard tests that allow one to quantify the image degradation that distortion causes. I'll see if i can add your shader to it and see what happens.

FWIW - for image quality comparisons it's better if you leave the chromatic correction off.

Out of curiosity, what are the conditions for your baseline image? It looks like it has texture filtering entirely disabled since I can't see any of the blurring that is endemic to bilinear filtering, but I can't see any of the "seams" typical of having texture filtering disabled either.

Out of curiosity, what are the conditions for your baseline image? It looks like it has texture filtering entirely disabled since I can't see any of the blurring that is endemic to bilinear filtering, but I can't see any of the "seams" typical of having texture filtering disabled either.

Everything is the same as in the standard Oculus World Demo. All I changed was the Shader code posted above and the uniforms needed to run it. You can toggle between them in the Demo posted above (C Key).

"TeraBit" wrote:

Although the warp concentrates more pixels in the middle of your vision, you don't actually see them, because to fit them in the middle, you are essentially skipping pixels.
...
The result is that the closer to the middle you are, the larger the sample size (taking more pixels in the original image into account), the further out, the closer to 1:1 pixel samples you get.

This seems a bit shaky to me. The distortion maintains a (roughly) 1:1 pixel correspondence inside the center of the image. This is because the warp is effectively curving the sampled texture away from you - the center suffers minimal distortion, but as the distortion increases as you follow the curve the space between subsequent samples increases. Thus, the areas where distortion has the worst impact on image quality is the edges and not the center.

Ideally, you need to be using less filtering towards the center and more towards the edges.

"TeraBit" wrote:

Everything is the same as in the standard Oculus World Demo. All I changed was the Shader code posted above and the uniforms needed to run it. You can toggle between them in the Demo posted above (C Key).

Cool. I'll dig through it when I have some time. Any chance you'd be willing to post the source code?

This seems a bit shaky to me. The distortion maintains a (roughly) 1:1 pixel correspondence inside the center of the image. This is because the warp is effectively curving the sampled texture away from you - the center suffers minimal distortion, but as the distortion increases as you follow the curve the space between subsequent samples increases.

You are right. I've got it backwards. It still works because I'm measuring the distance between surrounding texels to give the sample range. It's just doing the opposite of what I thought it was doing! :lol:

This is likely because the last warp shader I was doing was in Cuda, on my ray tracing demo. Thought somehow texture sample did the opposite of rays.. :roll:

In fact the middle seems to larger than 1:1 judging by just looking at an unwarped vs warped image. Hmmm what am I doing? :? Heh, still seems to work though.. 8-) Theories, you gotta love em.

"TeraBit" wrote:
Hmmm what am I doing? :? Heh, still seems to work though.. 8-) Theories, you gotta love em.

Sounds like it might just be a manual implementation of multi-sampling.

Actually, there seem to be a lot less subjective viewpoints being bandied around than I expected. :shock:

Has anyone actually tried the demo in the rift to see if they actually like the effect?
Regardless of how it (does or probably doesn't) work? ;)