Asynchronous timewarp example

This is driver/opengl limitation. You cant call opengl functions from multiple threads, at least not to same rendering context. Rendering context can be active/current in only on thread at a time ( usually the render thread ). Offcourse it is possible to create multiple context, but that introduces its own problems ( sharing resources etc. ).

His use of a common framework (and spelling mistakes) makes it difficult to follow, but it seems like he is creating a second off-screen window for the normal drawing thread and then creating a context from that window, and using the normal window and normal context for the asynchronous timewarp thread.
Which makes it considerably more difficult than I had hoped. Oh well.

"2EyeGuy" wrote:
I notice you are violating this rule:

"Oculus_Developer_Guide.pdf" wrote:
All of the following calls must be done on the render thread. This is the thread used by the application to create the main rendering device. ovrHmd_BeginFrame (or ovrHmd_BeginFrameTiming and ovrHmd_EndFrame, ovrHmd_GetEyePose, ovrHmd_GetEyeTimewarpMatrices.

In practice it's not actually necessary to call GetEyePose from the same thread as BeginFrame & EndFrame. What's required is that you only call GetEyePose in the interval between those two functions being called (granted. My example ensures this by having the distortion thread own a lock that it only releases during the interval between begin and end, and the rendering thread must acquire the lock in order to get the pose. Alternatively, I could fetch the eye poses on the main thread and use the lock to ensure the data was unchanged while copying across the thread boundary.

"2EyeGuy" wrote:
His use of a common framework (and spelling mistakes) makes it difficult to follow, but it seems like he is creating a second off-screen window for the normal drawing thread and then creating a context from that window, and using the normal window and normal context for the asynchronous timewarp thread.
Which makes it considerably more difficult than I had hoped. Oh well.

This is the mechanism used to create shared contexts in glfw. Any mechanism will do actually. You just have to have one context for the rendering thread and one for the distortion thread. The only shared resource are the textures used to hold the rendered scene.

WOW! I just tried an extremely simple alternative (in fact, I only implemented it this morning on the bus on my way to work):

"Pseudo-Asynchronous Timewarp" (well, it does not really need a name, but it's not regular timewarp and it is not really asynchronous).

The simple solution is to create a simple function, which checks if there is enough time left for the current frame and if not, it just feeds the last buffer to the ovr framebuffer, calls EndFrame and then BeginFrame again. In my case it looks like this:

Edit: Dear dev, who stumbles upon this: Do not use this code. It's not a good idea and on top of it, the time calculations are complete non-sense. Have a look at jherico's example instead.

void pseudoAsyncTimewarp() {
	ovrFrameTiming t = ovrHmd_GetFrameTiming(hmd, 0);
	if (t.ThisFrameSeconds + t.DeltaSeconds - t.TimewarpPointSeconds > -currentConfig.patTime/1000.) {

		/* WRITE OLD FRAME TO OVR FRAMEBUFFER */

		ovrHmd_EndFrame(hmd, lastHeadPose, &EyeTexture[0].Texture);
		ovrHmd_BeginFrame(hmd, 0);
	}
}

(edit: /* WRITE OLD FRAME TO OVR FRAMEBUFFER */ of course has to be replaced with appropiate code specific to the engine.)

currentConfig.patTime is a config parameter to adjust the time that may be left in the current frame before the engine wusses out and recycles the last frame.

I just call this code between all major tasks in my render pipeline (i.e. Render geometry, pseudoAsyncTimewarp, render shadows, pseudoAsyncTimewarp, render reflections, pseudoAsyncTimewarp, render deferred shader, pseudoAsyncTimewarp, ...) As I have a quite simple and hence fast final color mapping step, I have the last frame left in a buffer anyway and can just do the color mapping again if I want to repeat the last frame. As most frames should be ready on vsync anyway, I would expect this to be cheaper on few frames, which missed the deadline than copying the whole framebuffer on every single frame - but I have to benchmark this...

Also note, that I provide "lastHeadPose" to EndFrame, which holds the pose, which matches the last frame...

The result is amazing! I just deliberately increased quality settings and the frame rate dropped to 30-40fps - and I still did not notice any judder - before this I was still struggling to optimize my engine to hit 75 fps and did not like the judder at 65-70fps - with this technique, however, now I am so amazed, that I have to post here right away :)

So, please, try it yourself and let's discuss optimizations and drawbacks! I think, this is a very simple implementation, which might help until true asynchronous timewarp is easily available. Maybe one should consider a way to estimate the perfect time before vsync to give up?

(Of course, you can also wait a week or two until my demo is available, in which this will certainly be an option to play with :D )

"DiCon" wrote:
The simple solution is to create a simple function, which checks if there is enough time left for the current frame and if not, it just feeds the last buffer to the ovr framebuffer, calls EndFrame and then BeginFrame again.
...
I just call this code between all major tasks in my render pipeline (i.e. Render geometry, pseudoAsyncTimewarp, render shadows, pseudoAsyncTimewarp, render reflections, pseudoAsyncTimewarp, render deferred shader, pseudoAsyncTimewarp, ...)

This appraoch can work, but there are serious architectural and maintenance issues with it. Basically it requires every piece of the code inside the rendering engine to be able to estimate it's runtime in advance and/or parcel it's work into smaller sections. This leads to premature optimization and potentially the injection of many calls to pseudoAsyncTimewarp where they might never be needed, leading to additional maintenance costs and potentially performance degradation as pseudoAsyncTimewarp gets called many many times without triggering a timewarp frame.

It also has issues if you have the possibility of expensive tasks that can't be subdivided, say for instance a final lighting step in a deferred shading pipeline, or rendering code in a third party library. It also requires that every piece of code in the rendering pipeline have special knowledge of this one function that doesn't .

The approach is very reminiscent of the 'Yeild' technique in the old Win16 API. Back before Win32, Windows didn't have true multi-tasking. As long as an application was processing a given message, everything else on the system was blocked. The solution at the time was the Yield function. The idea was that if you were going to be executing a long running task that might end up causing the UI to lock for a significant amount of time, you would call Yield inside your tasks so that windows could process other messages for other applications. This solution suffered the same drawbacks of your pseudoAsync timewarp. It placed additional burden on components to make special considerations that were outside the scope of whatever they were trying to do. If I'm writing a piece of code intended to find a prime number with at least N digits, the last thing I'm going to want to think about is where I should insert a special function call designed to let the rest of the operating system update the UI.

Ultimately, while these techniques do work, and are perhaps even very effective, as the complexity of your codebase grows, the maintenance cost for using them grows as well, acting like a technique tax. For very complex systems, the associated cost ends up being untenable. By comparison, true multi-threading (either at the OS message handling level as in Win32, or at the rendering system level as in OpenGL) tends to involve a one-time cost of getting it right. Once you've done it, there's no longer much or any incremental cost as the size of your system grows.

ok, I understand and agree with every single point. Nevertheless, for a simple demo I prefer this quick and dirty approach over not doing anything about low framerates...

On the other hand, I am already successfully sharing an OpenGL context over multiple threads during my loading screen - maybe I should look into your proper approach using multipe context. Damn, I was so excited about these few lines, which seemed to solve all my problems ;)

"jherico" wrote:
This is the mechanism used to create shared contexts in glfw. Any mechanism will do actually. You just have to have one context for the rendering thread and one for the distortion thread. The only shared resource are the textures used to hold the rendered scene.

How are you sharing the textures used to hold the rendered scene?

"2EyeGuy" wrote:

"jherico" wrote:
This is the mechanism used to create shared contexts in glfw. Any mechanism will do actually. You just have to have one context for the rendering thread and one for the distortion thread. The only shared resource are the textures used to hold the rendered scene.

How are you sharing the textures used to hold the rendered scene?

If you are using glfw, you can use the "share" parameter:

GLFWwindow * window = glfwCreateWindow( scrWidth, scrHeight, "Worker", NULL, NULL);
GLFWwindow * windowOVR = glfwCreateWindow( scrWidth, scrHeight, "Display", glfwGetPrimaryMonitor(), window);

The very last parameter of the second line tells glfw, that windowOVR should share "everything" (actually, not everything - there seem to be some details like FBOs) with "window". You can simply use texture IDs from one context in the other one. Just make sure to make the right context current using

glfwMakeContextCurrent(windowOVR);

Also you might want to hide the background window with

glfwWindowHint(GLFW_VISIBLE, GL_FALSE);

This is just what I found out, trying to implement the "true asynchronous timewarp" without my Rift at hand... I will report back, when I tried it and might have encountered some more crucial details...

@jherico:
Thanks again, for pointing out the flaws in my approach. So far using a second context does not seem to be hard to implement after all. I feel like such a fool...
Just one question: Are there some race conditions, that I should be aware of? Currently, I just use a simple mutex to prevent copying the render result to the Oculus-framebuffer, which so far I only disallow if a new result is beeing written to the texture until the variable holding the corresponding head pose has been updated right after this.

EDIT: Just saw from your example, that I should secure getPose against being called between EndFrame and BeginFrame. Makes sense ;)

"DiCon" wrote:
Thanks again, for pointing out the flaws in my approach. So far using a second context does not seem to be hard to implement after all. I feel like such a fool...

Don't stress. Like I said, it is a reasonable approach for smaller projects where you're just testing out stuff. Of course if you expect your project to grow over time, then the earlier you switch to true async the better.

"DiCon" wrote:
Just one question: Are there some race conditions, that I should be aware of? Currently, I just use a simple mutex to prevent copying the render result to the Oculus-framebuffer, which so far I only disallow if a new result is beeing written to the texture until the variable holding the corresponding head pose has been updated right after this.

EDIT: Just saw from your example, that I should secure getPose against being called between EndFrame and BeginFrame. Makes sense ;)

There are a couple of synchronization issues. The first, as you saw was that GetEyePose can only be called between BeginFrame and EndFrame. The easiest way to do this is to have the begin/end thread own a lock which it only releases between the two calls. In order to give the render thread plenty of time to fetch the eye poses, it then executes a wait until just before the distortion time. Most of the OVR distortion thread time will likely be spent in this wait state, since the distortion takes very little time.

Another sync issue is that you need to make sure you pass the eye pose for a given frame and the texture which has been rendered with that frame over to the distortion thread atomically. If the distortion thread picks up one but not the other during a given render, you will get judder or flickering.

Also, you want to make sure that the rendering has actually completed before you send it over to the distortion thread. Simply calling the OpenGL functions doesn't mean the rendering is done because of the inherently asynchronous nature of OpenGL (you're really dealing with at 3-5 threads... our two explicit threads, the OpenGL driver side thread, for which there is probably an instance for each context, and the actual GPU rendering thread, but fortunately you don't need to think about most of this complexity). Additionally, calling glFinish should NOT be used to wait on the rendering commands to complete. If you're waiting for the left eye to finish, you're basically creating a pipeline stall that you could be filling with commands for the right eye. Instead, we use glFenceSync to create a sync object. The sync object gets signaled when it is processed by the GPU, meaning that all the commands prior to it have been executed. We're able to query the glFenceSync at any time to find out whether the render has completed, and it if has, we send the pose and the completed texture over to the distortion thread.

"jherico" wrote:
Also, you want to make sure that the rendering has actually completed before you send it over to the distortion thread. Simply calling the OpenGL functions doesn't mean the rendering is done because of the inherently asynchronous nature of OpenGL (you're really dealing with at 3-5 threads... our two explicit threads, the OpenGL driver side thread, for which there is probably an instance for each context, and the actual GPU rendering thread, but fortunately you don't need to think about most of this complexity). Additionally, calling glFinish should NOT be used to wait on the rendering commands to complete. If you're waiting for the left eye to finish, you're basically creating a pipeline stall that you could be filling with commands for the right eye. Instead, we use glFenceSync to create a sync object. The sync object gets signaled when it is processed by the GPU, meaning that all the commands prior to it have been executed. We're able to query the glFenceSync at any time to find out whether the render has completed, and it if has, we send the pose and the completed texture over to the distortion thread.

At this point I seem to have run in a problem, I do not fully understand as I never used OpenGL with multiple contexts. I have implemented asynchronous timewarp with glFenceSync as you suggested, but I see in your example, that glClientWaitSync is called on the main thread, which never continues rendering unless an eye has been finished by using

if (0 != eyeFences[eye]) {
          continue;
}

This seems to me quite close to a plain glFinish - In your case the "other" eye can still be rendered, but if I do this for an engine, with multiple render steps, waiting for the result either means a stall or I have to do this so late in my code, that I would not use the new image immediately and get additional latency. Therefore I tried to place glClientWaitSync in the other thread just before EndFrame and stumbled onto the following problem:

As soon as I place a GL command into the second thread, which would put load on the GPU (i.e. not just a state change), the second thread syncs with the first thread. So, when I just place a cout << "Test" into the second thread, it will flood my terminal as expected. But as soon as EndFrame is in there as well, I get exactly one "Test" output per frame rendered on the other thread. This happens even if I do not use glClientWaitSync - literally with just BeginFrame, cout and EndFrame in there. I also noticed that on the second thread glClientWaitSync does not return immediately, regardless of the chosen timeout or if I leave out the flush-flag. I even replaced it by glGetSynciv, which also did not return immediately.

(Edited from here, after giving up on a wild theory about a common command queue for both contexts)

On my loading screen I have no such problems. Here the rendering thread runs independently from the loading thread, no matter what. It seems like EndFrame has to wait for something on the rendering thread, but there is no glFlush or glFinish in my code - I do all readbacks using PBOs. Could there be a problem if some commands from the rendering thread are too huge chunks, which block the GPU for commands from the second context? I use cascaded shadow maps and can render almost the whole scene in a single call with an geometry shader, that splits the vertices to the individual shadow maps - I could imagine, that the GPU cannot handle any command during this operation...?