Asynchronous timewarp example

"DiCon" wrote:
At this point I seem to have run in a problem, I do not fully understand as I never used OpenGL with multiple contexts. I have implemented asynchronous timewarp with glFenceSync as you suggested, but I see in your example, that glClientWaitSync is called on the main thread, which never continues rendering unless an eye has been finished by using

if (0 != eyeFences[eye]) {
          continue;
}

This seems to me quite close to a plain glFinish - In your case the "other" eye can still be rendered, but if I do this for an engine, with multiple render steps, waiting for the result either means a stall or I have to do this so late in my code, that I would not use the new image immediately and get additional latency.

To be sure, this was just an example I whipped out in a single day and there are better approaches to this. For instance, I could implement something like triple buffering, where as soon as I complete both eye renders, I start on the next renders. The downside to this (aside from being more complex) is that I'd need additional offscreen textures to store the new images. In addition, if I'm capable of four eyes per frame instead of two, then although I'm not stalling the pipeline, I am filling it with junk, because there's no value in the first renders per eye... they're never seen.

If on the other hand I'm capable of rendering somewhere between 2 and 4 eyes per frame, then what happens is that my additional rendered eyes frequently cross the vsync boundary. This means that you're increasing the average amount of time that timewarp has to compensate for, which increases artifacts.

The problem is that you have two competing concerns, maintaining exactly 75 frames per second, and keeping the GPU pipeline full. You can't really satisfy both at the same time.

"DiCon" wrote:
Therefore I tried to place glClientWaitSync in the other thread just before EndFrame and stumbled onto the following problem:

As soon as I place a GL command into the second thread, which would put load on the GPU (i.e. not just a state change), the second thread syncs with the first thread.

You'll probably find that glClientWaitSync on the other thread is returning 0. Fence sync objects can't be used across the CPU threads. They're tied to the OpenGL context in which they were created.

OK, have to take my son to school, so might reply more later.

"jherico" wrote:
The downside to this (aside from being more complex) is that I'd need additional offscreen textures to store the new images. In addition, if I'm capable of four eyes per frame instead of two, then although I'm not stalling the pipeline, I am filling it with junk, because there's no value in the first renders per eye... they're never seen.

If on the other hand I'm capable of rendering somewhere between 2 and 4 eyes per frame, then what happens is that my additional rendered eyes frequently cross the vsync boundary. This means that you're increasing the average amount of time that timewarp has to compensate for, which increases artifacts.

So to elaborate.... you could discard the images rendered from the previous eye once you'd gotten to the next frame, but then you're likely consuming some of your current rendering time budget with the discarded previous eye.

At 75 Hz or (maybe even 90 Hz for the consumer version?) your rendering time budget is so constrained that keeping the pipeline full probably isn't going to be high on the list of problems. You want to keep it full because that means you're using it most efficiently, but since you're blocked on rendering a given scene until you have the head pose, and that scene has an effective expire date of the next v-sync, populating the pipeline efficiently is very hard to do.

"jherico" wrote:
You'll probably find that glClientWaitSync on the other thread is returning 0. Fence sync objects can't be used across the CPU threads. They're tied to the OpenGL context in which they were created.

Nope, it definitely returns different results as my checks for GL_ALREADY_SIGNALED or GL_CONDITION_SATISFIED​ succeed half the time... But if it's not supposed to work, this may be undefined behavior and the reason for the long time until it returns. And maybe, the slowdown of the second thread had multiple reasons...

So, following your thoughts about a full pipeline not being top priority, I will now try to do the glClientWaitSync for the result just before getting the next pose... And then spend the rest of the evening figuring out, what's still wrong :D

Greetings.

I'll just chime in and say that OculusWorldDemo (OWD) has a "somewhat hidden" feature that allows simulating what async-time-warp (ATW) would look like when it's implemented as intended.

In OWD, hit 'Tab', and under Timewarp, modify the RenderIntervalSeconds value to drop the simulated 3D render down to something less than 75 fps. If you go all the way down to 10 fps, you'll see that looking around is still fairly smooth, and you just get more black on the sides, that will be filled up as the new scene render comes into view. Moving your avatar through the scene, or having you head move side-to-side will show the artifacts of the low-frame-rate as timewarp is only accounting for orientation and not positional changes.

Again, this is just a simulation of ATW, and the implementation in OWD, is not to be taken as reference.

Another cool trick is to completely freeze the rendering of the scene by hitting 'C', and looking around. You'll see that tracking keeps working fine even though the 3D scene is no longer being rendered.

I did it.

Besides trying to wait for the fence on the wrong thread, it was quite a silly mistake that messed up the version in which I waited on the same thread:

I should have read the docs more thoroughly as I did not realize, that the timeout is to be set in nanoseconds. (Nanoseconds! Really?*) So I set it in milliseconds instead and most of the time, glClientWaitSync would timeout, which in turn means, that the render targets were not swapped. Therefore, the engine would render directly to the same texture, that EndFrame used on the second thread and (from here I am guessing) thus it had to wait for the first thread to finish.

Am I right about the last bit? It should not be possible to access a texture in one context, while another context is actively rendering to it..? (Though I am happy, that it's working now, I'd really like to understand the problem here...)

*At work I control the timing of an experiment on a (sub)picosecond time scale by changing the distance which a laser pulse has to travel. If I wanted to change the timing by 1 nanosecond, I would move a retroreflector by 15cm. That's what a bloody nanosecond is. The time it takes the light itself to travel whopping 30cm. - Of course I know, that the timeout of glClientWaitSync will not have that precision, but... but... I just didn't expect nanoseconds here... :geek:

"DiCon" wrote:
I should have read the docs more thoroughly as I did not realize, that the timeout is to be set in nanoseconds. (Nanoseconds! Really?*) So I set it in milliseconds instead and most of the time, glClientWaitSync would timeout, which in turn means, that the render targets were not swapped. Therefore, the engine would render directly to the same texture, that EndFrame used on the second thread and (from here I am guessing) thus it had to wait for the first thread to finish.

Generally you don't want to set a timeout when calling glClientWaitSync. Instead you just set the timeout to 0 and do something specific if the result indicates that the image has been rendered.

"DiCon" wrote:
Am I right about the last bit? It should not be possible to access a texture in one context, while another context is actively rendering to it..? (Though I am happy, that it's working now, I'd really like to understand the problem here...)

No, you should not render a texture that is being drawn to. However, it's possible that you're only setting up a situation where that's possible, but that it doesn't necessarily happen. Keep in mind the SDK performs it's own wait until just prior to the vsync to actually execute the distortion, which may given enough time for the texture rendering to complete.

"jherico" wrote:
Generally you don't want to set a timeout when calling glClientWaitSync. Instead you just set the timeout to 0 and do something specific if the result indicates that the image has been rendered.

ok, here are my thoughts on this:

That's what I did first, but I would have to call it repeatedly until it returns that the render is ready. As this has to be done on the same context and switching the context to a second thread requires the first one to actively give up the context, I cannot check asynchronously. So, I am stuck with two options (unless I don't see the golden third way):
a) not doing anything while checking in a loop
b) checking at specific points in my code inbetween other tasks (similar to the pseudo-thing I proposed before)

a) is not much different from what I am doing now with a long timeout. I would think that calling a function a single time and waiting for its return should be preferred over calling a function repeatedly until the same result occurs...?

b) should not help much as well, because if glClientWaitSync reports that I should do something else, the CPU will request the pose for one eye, create a projection matrix, load a VAO and initiate the next render. There is nothing left to do for the CPU at this point... But indeed, if I just wait at glClientWaitSync, the GPU would stall during these steps until the next render is issued, which in this case would be a single call to draw about 5000 buildings at which point the engine is GPU-limited again. So I would expect b) to only save the time of getting the pose and doing a few matrix multiplications.

The problem with this little gain is, that the fresh image is rotting in a texture and not being used by timewarp until I check glClientWaitSync again. In the end this is just adding latency as well and it should not matter if I waste time at the beginning of a frame or after finishing it before I actually present the frame. So to actually gain from this method I would have to reduce the average time between two checks below the time I have gained - which means inserting a function call to the glClientWaitSync-part every few lines...

Therefore, as I do not see a fps-drop in my method and do not find the theoretical gain of a few µs worth obfuscating my code I went for a) - but with a timeout instead of multiple function calls... So what's c)? ;)

(I hope, it's ok, that I am bugging you with these details - I am really grateful as I am learning a lot through this discussion - thanks for that!)

"DiCon" wrote:
If you are using glfw

I'm not. I'm using the Windows API, and whatever functions are available in Dolphin.

I'm slowly making progress. I seem to have my threads and contexts working correctly. Now I just need to add the asynchronous rendering itself.

EDIT:

// Grab the most recent textures
    for_each_eye([&](ovrEyeType eye) {
      ((ovrGLTexture&)(eyeTextures[eye])).OGL.TexId =
        textureIds[eye];
    });

    ovrLock.lock();

Shouldn't the ovrLock.lock() come before the "Grab the most recent textures part"?

It works!
Dolphin finally has asynchronous timewarp!

It's not quite as good as I had hoped, I might not be getting 75 FPS even with asynchronous timewarp, or perhaps timewarp just isn't that great. But it is working!

I might need to improve head tracking prediction. Currently, like your code, I am getting the predicted eye pose for the wrong frame which may be a few frames earlier than when the real frame I am working on will be ready. Any idea how best to get a predicted eye pose, or estimate when the frame will really be ready?

I also have no way of evaluating its performance yet or testing latency.

I was getting crashes on AMD cards. The problem was a race condition where sometimes the asynchronous timewarp thread would try to read the textures that the other thread had just created, but OpenGL hadn't finished creating them yet. The trick was calling glFinish after initialising the main thread.

That isn't a problem with your code, because you initialise things like that in the same thread that does asynchronous timewarp.