Skip to main content
Up and Coming
April 11, 2013

Shader Optimization.

I was playing with the shader and wondered if it could be optimized.

Lets start with the standard Shader


float2 HmdWarp(float2 in01)
{
float2 theta = (in01 - LensCenter) * ScaleIn; // Scales to [-1, 1]
float rSq= theta.x * theta.x + theta.y * theta.y;
float2 theta1 = theta * (HmdWarpParam.x + HmdWarpParam.y * rSq +
HmdWarpParam.z * rSq * rSq + HmdWarpParam.w * rSq * rSq * rSq);
return LensCenter + Scale * theta1;
}

float4 main( float2 texCoord : TEXCOORD0,float4 color : COLOR0 ) : SV_Target
{
float2 tc = HmdWarp(texCoord);
if (any(clamp(tc, ScreenCenter-float2(0.25,0.5), ScreenCenter+float2(0.25, 0.5)) - tc))
return 0;
return Texture.Sample(Linear, tc);

};


That's 11 Multiplies, 6 Additions and 3 Subtractions per pixel. (eyeballed the numbers sorry, could be off by one or two) and on my GTX670 takes 0.1447264 milliseconds to execute (based on averaging the time of a 100 frames of just the draw with the shader)

Which could be good, could be bad, nothing to compare it to, so lets compare it to a shader that does not warp and just samples the texture.


float4 pixelShader( float2 texCoord : TEXCOORD0,float4 color : COLOR0 ) : SV_Target
{
float4 c = Texture.Sample(Linear, texCoord);
return c;
}


which takes 0.0272857 milliseconds to execute (again GTX670, 100 frames)

So the warping shader is 5.3 times slower than just bare sampling.

There must be a better way, why are we recalculating the distortion map every frame single frame? seems rather wasteful. Lets render the distortion map to texture instead. We'll use the Red channel for the X distortion, and the Green channel for the Y distortion. If we make a texture of format R32G32_Float it should fit our needs nicely. And the upside is we only have to do the nasty math once!

our shader now looks like this


float2 HmdWarp(float2 in01)
{
/// Same as above, cut to keep the post size down.
}

float4 main( float2 texCoord : TEXCOORD0,float4 color : COLOR0 ) : SV_Target
{
float2 tc = HmdWarp(texCoord);
if (any(clamp(tc, ScreenCenter-float2(0.25,0.5), ScreenCenter+float2(0.25, 0.5)) - tc))
return 0;
return float4(tc.x, tc.y,0,0);
};


The resulting texture will look kinda like this (Used bad parameters, had to guess since I don't have a rift yet so the horizontal mapping is somewhat off)



Sweet, almost in business! lets update the warping shader to make use of the new texture. First of all the linear sampler has to go, a MinMagMipPoint sampler will fit our needs better.


SamplerState PointSampler : register(s0);
Texture2D Texture : register(t0);
Texture2D TextureWarp : register(t1);

float4 main( float2 texCoord : TEXCOORD0,float4 color : COLOR0 ) : SV_Target
{
float4 c = TextureWarp.Sample(PointSampler, texCoord);
float2 RealCoord = float2(c.r,c.g);
return Texture.Sample(PointSampler, RealCoord);
};


Ahh much simpler now! New execution time 0.0598547 milliseconds and no visual difference! 2.5 times speed up from the original shader! Not bad for 20 minutes of tinkering.

    80 replies

    Participant
    April 25, 2013
    struct VS_Output
    {
    float4 Pos : SV_POSITION;
    float2 Tex : TEXCOORD0;
    };

    VS_Output VS(uint id : SV_VertexID)
    {
    VS_Output Output;
    Output.Tex = float2((id << 1) & 2, id & 2);
    Output.Pos = float4(Output.Tex * float2(2,-2) + float2(-1,1), 0, 1);
    return Output;
    }


    A neat little vertex shader used in Timothy Lottes' GL FXXA code. Should cut the render time down even further for the warp shader as the buffers the oculus sdk feeds in for the fullscreen quad, vertex UV / Pos, are no longer required. Better still, you could also render using a triangle list if the state change does not stall your render pipeline.
    Up and Coming
    April 25, 2013
    gallantpigeon, I have doubts that would speed anything up, as the vertex setup is negligible in this case... however, I really like that trick, as it's silly to describe the screen with a bunch of vertex attribs and load them!

    I never knew about vertex_id being available, and now I see it was since GLSL 1.3... it's tough to keep up with all changes. Edit: correction, it's always been available; the first document I checked was misleading.
    Up and Coming
    April 27, 2013
    Here's the barrelwarp distortion shader plus chromatic aberration correction from the v0.2.1 SDK, let's see how well we can optimize three dependent texture fetches :P
    sampler2D readTexture : register(s0);
    float2 LensCenter;
    float2 ScreenCenter;
    float2 Scale;
    float2 ScaleIn;
    float4 HmdWarpParam;
    float4 ChromAbParam;

    // Scales input texture coordinates for distortion.
    // ScaleIn maps texture coordinates to Scales to ([-1, 1]), although top/bottom will be
    // larger due to aspect ratio.
    float4 main(in float4 oPosition : SV_Position, in float4 oColor : COLOR,
    in float2 oTexCoord : TEXCOORD0) : SV_Target
    {
    float2 theta = (oTexCoord - LensCenter) * ScaleIn; // Scales to [-1, 1]
    float rSq = theta.x * theta.x + theta.y * theta.y;
    float2 theta1 = theta * (HmdWarpParam.x + HmdWarpParam.y * rSq + HmdWarpParam.z * rSq * rSq + HmdWarpParam.w * rSq * rSq * rSq);

    // Detect whether blue texture coordinates are out of range since these will scaled out the furthest.
    float2 thetaBlue = theta1 * (ChromAbParam.z + ChromAbParam.w * rSq);
    float2 tcBlue = LensCenter + Scale * thetaBlue;
    if (any(clamp(tcBlue, ScreenCenter-float2(0.25,0.5), ScreenCenter+float2(0.25, 0.5)) - tcBlue))
    return 0;

    // Now do blue texture lookup.
    float blue = tex2D(readTexture, tcBlue).b;

    // Do green lookup (no scaling).
    float2 tcGreen = LensCenter + Scale * theta1;
    float4 greenColor = tex2D(readTexture, tcGreen);
    float green = greenColor.g;
    float alpha = greenColor.a;

    // Do red scale and lookup.
    float2 thetaRed = theta1 * (ChromAbParam.x + ChromAbParam.y * rSq);
    float2 tcRed = LensCenter + Scale * thetaRed;
    float red = tex2D(readTexture, tcRed).r;

    return float4(red, green, blue, alpha);
    };


    Edit: The stock v0.2.1 SDK comes with a D3D10-style shader. Mine is in the D3D9 style that is more similar to classic HLSL and cg than newer HLSL.
    Up and Coming
    April 29, 2013
    I was wondering if there were benchmarks that showed if a texture lookup was better than recomputing and it was here in the other forum section all along.

    Optimize all the shaders.


    As a programming language researcher this makes me sad.
    (╯°□°)╯︵┻━┻
    Up and Coming
    May 4, 2013
    "jwilkins" wrote:
    I was wondering if there were benchmarks that showed if a texture lookup was better than recomputing and it was here in the other forum section all along.

    Optimize all the shaders.


    As a programming language researcher this makes me sad.

    This is an incredibly in-depth (unless you're a graphics driver author) description of how modern graphics cards process commands from software through the hardware into pixels on your screen, as written by a smarter guy than me:
    http://fgiesen.wordpress.com/2011/07/09/a-trip-through-the-graphics-pipeline-2011-index/

    I'd recommend reading that (it's likely the best explanation of "how do graphics cards *really* work" that I've been able to find anywhere online), but in my understanding the answer is "it depends". There's several factors behind why it would depend, but the most pertinent is that modern GPUs perform texture lookups at a different speed than they perform ALU ops. This turns out to be because of many different reasons (texture unit to arithmetic unit ratios on the silicon, texture memory bandwidth and cache size, cache access patterns, ALU vectorization, certain GPU architectures handle dependent texture fetching differently than predicted texture fetches, etc.). Take a look at the relative disparity of core configuration ratios even within the same generation and architecture of GPU from the same vendor:
    http://en.wikipedia.org/wiki/Comparison_of_Nvidia_graphics_processing_units#GeForce_600_Series
    Up and Coming
    May 4, 2013
    I think I should explain what "makes me sad" in more detail.

    I understand the need for all of the optimizations presented in that GDC presentation. In fact I'd argue that many of them are not optimizations at all, but just the correct way to write IEEE floating point code.

    (I do question the accuracy of some of the information, does GLSL and HLSL really say that expressions are evaluated left-to-right? That is more restrictive than C/C++ and ties the compiler's hands!)

    So what makes me sad? I find high level IEEE floating point expressions to be a very leaky abstraction that only barely covers up the assembly language it compiles to. The main problem is that you are mixing up what you want to compute with how to compute it, where to compute it, and in what order to compute it.

    My research is into how we can write this kind of code but express potentially optimizing transformations separately from what we want to compute. More plainly, this means you can write down the mathematical expression for what you want to compute in one place and you would not have to change it whether or not you wanted to recompute it each time or pre-compute it and store it in a texture. Right now making this transformation requires writing two completely different programs.

    See MIT CSAIL Lab's Halide language for one example of how this might work.

    I'm glossing over a lot of details and since it is probably too off topic to discuss this further I invite anybody who has any questions to PM me.
    (╯°□°)╯︵┻━┻
    Up and Coming
    May 12, 2013
    On the topic of optimization...

    Given that there are many pixels that aren't really visible on the Rift would it make sense to have a mask you could write to the z-buffer or a scissor to eliminate unnecessary drawing? Especially when rendering at higher resolutions to improve quality.

    Currently we have...
    7" 1280x800 panel = 1024000 pixels
    It's aproximately a 250 pixel radius visible circle per eye = 196349 pixels per eye
    392698 pixels visible for both eyes
    392698 / 1024000 = Only 38% (approximately) of the panel's pixels are ever visible.

    In order for this to be accurate though it would have to be the equivalent of what that 250px radius circle would look like "pre-warped". Not sure what that would come to exactly. I believe it is somewhere between the 250px radius circle and a 500px square. Future versions of the Rift will likely have less non-visible pixels but even with optimal display panel size there will still be some non-visible pixels when viewing through round openings.

    This image approximately illustrates the post-warp pixels that end up being visible through the Rifts optics with A-cup lenses and the assembly extended all the away from the screen. Try opening it full screen and viewing it through the Rift, you should not be able to see much, if any of the darkened areas. Since the darkened areas represent pixels that are never visible to the user time spent rasterizing these (or rather the pixels they come from pre-warp) is wasted.
    Inspiring
    May 12, 2013
    Ideally this should be dependent on the lens in use (a/b/c).
    Up and Coming
    May 12, 2013
    "tmek" wrote:
    Given that there are many pixels that aren't really visible on the Rift would it make sense to have a mask you could write to the z-buffer or a scissor to eliminate unnecessary drawing? Especially when rendering at higher resolutions to improve quality.
    ...
    This image approximately illustrates the post-warp pixels that end up being visible through the Rifts optics with A-cup lenses and the assembly extended all the away from the screen. Try opening it full screen and viewing it through the Rift, you should not be able to see much, if any of the darkened areas. Since the darkened areas represent pixels that are never visible to the user time spent rasterizing these (or rather the pixels they come from pre-warp) is wasted.

    Okay, seeing as you posted this image to two threads, I will respond in both threads as well.

    The image above shows the difference between eyes close to the lenses and eyes farthest from the lenses, depending on the Rift DK "coin slot" adjustment settings.

    Like you said in your other thread, "It shows what you see using A-cup with the assembly extended all the way back." But if you adjust it so your eyelashes brush the lenses, when looking straight ahead I can actually see the edges of the screen in ALL DIRECTIONS, which means typical Rift pre-warp algorithms are showing me some black pixels where they COULD show me useful content. So I can actually see slightly more content than the outer dimmed portion of the image pair shown above (including a small amount of the screen bezel). Rotating my eyeballs to actually look directly at the screen edges reduces my FoV in that direction, so I only see the screen edges in my peripheral vision while looking straight forward.

    The image in this thread is showing what you may see while wearing eyeglasses, or when you adjust it according to the HL2 configuration instructions. I very much prefer the much larger FoV from having the "assembly RETRACTED all the way" with my eyelashes brushing the lenses.

    Beware that the shorter B and C cups give a smaller FoV than the longer A cups, and moving your eyes farther from the lenses gives a MUCH smaller FoV.

    I can already see black pixels surrounding the content, so please do NOT encourage developers to waste even more of my FoV! You may choose to sacrifice most of you FoV by placing your eyes far from the lenses, but I do not. I want my huge FoV, which is exactly what makes the Rift design vastly superior to its competition, IMHO! Your image posted to two threads, and your advice to sacrifice half of my FoV, just makes me sad...
    :roll:
    This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.
    Up and Coming
    May 12, 2013
    Hi Geekmaster,

    I can understand somewhat where you are coming from and I've read enough of your posts to know you are a very intelligent person. Respect. Also, In absolutely no way am I trying to get any FOV cut! However, I believe there is value in defining and understanding these ranges. At the very least I'd like to get a feel for what the ranges are and see if other people agree or learn what their ranges seem to be, or learn if there's something I haven't thought about yet.

    I've included an updated image. As I mentioned in the first image I used A-cups at the largest assembly distance. That's the inner circle. I do not wear glasses, although I usually like to keep it back so that my eyelashes never touch the lenses. For me that was pretty much all the way back.

    To try to find what the largest FOV attainable might be I dialed in the assembly closest to the panel and put my eyes as close to the lens as possible. I literally got Eyeball Juice(tm) on the lenses. The blue band represents the extents what I could see doing this. However I intentionally made it somewhat bigger than what I was able to see. I would say the practical best FOV range is about halfway through the dark blue circle region. "Practical" meaning you actually are wearing the Rift as intended not trying to see pixels you normally wouldn't by peeking through the lenses from a side angle you wouldn't be able to attain while actually wearing the headset in a proper fashion.

    I believe that the B and C cups give less FOV so they fall within these ranges as well (you seemed to agree with this also).

    You seemed to be saying that you could see "area beyond the content" or "the black pixels" I've colored that area magenta. There's almost nothing I can do to see those pixels. Even trying to mash my eyeballs right up on the lens, with a-cup, all the way in even from side angles. Again, I when wearing the Rift as intended I couldn't see further that about halfway through the dark blue circle radius. The only way I could see outside of the blue panel was to put my hands on the rift while wearing it and manipulate it. Anytime I saw outside of the blue band I would notice either my hands were on the rift moving it, or that I had it well off center. For example I might could see outside the blue band at the bottom but then I wasn't seeing any of the blue band at the top (being "cheaty", for the sake of this experiment).

    I would really be interested to know if while wearing the Rift in any reasonable manner and looking at this newer image:

    1.) Are you really able to see the magenta areas? (if so, can you describe the configuration, lens choice etc..)
    2.) Without being "cheaty" can you see outside of the dark blue band? (if so, approximately how many pixels extra would you estimate? and again what was your configuration?)