Timewarp theory

Awesome! I like your references to papers describing these techniques, dating back to the mid 1990's. :D

It seems there is almost always prior art, if we just expand our search terminology. Standing on the shoulders of giants, and all that implies.

If I have seen further it is by standing on ye sholders of Giants.
— Sir Isaac Newton

Oh yes - this stuff has been discussed for a long time - we're not pretending we invented it!

Two big caveats with the results of that paper are that there were no moving objects in the scene, and that (as far as I can tell) they were not using a "head model" that produced parallax from head rotation. These are actually kinda the same thing of course. Once you have either of these, the lower frame rates become fairly obvious.

Timewarp has basically three goals, and they're fairly independent:

1. To allow use of rolling-shutter HMD displays.
2. To allow a "late read" of the IMU and decrease latency from head motion.
3. To try to "upsample" from low framerate to high ones.

It does the first two well, but so far in all our experiments it does the last one very badly in anything but contrived conditions (no motion, no parallax - as in the paper). I know it would be awesome if the last thing worked, but being awesome and being possible are not the same thing!

We're still looking at it, and at more advanced timewarp methods that use the depth buffer, but right now none work well, and we still recommend everyone try as hard as they possibly can to hit the right framerate.

The head model to use in warping has interesting tradeoffs. I did actually used a global head model (7cm forward and 7cm up) for both scene rendering and warping, which yes can cause visibly jerky motion with low frame rates and near objects (particular the avatar torso), but it's still better than no warping.

From my cursory review of the Oculus method, it appears you only use the head model during rendering, but not during warping, which I believe introduces a similar error.

My current issue with warping is that it stalls the GPU too much, lowering frame throughput. It also doesn't play nice with a fixed frequency VSYNC and variable rendering frame rates, but that should be able to be avoided using technologies like G-SYNC.

"tomf" wrote:
We're still looking at it, and at more advanced timewarp methods that use the depth buffer, but right now none work well, and we still recommend everyone try as hard as they possibly can to hit the right framerate.

Doctor, it hurts when I move my arm like this. :roll: "Don't move your arm like that"

"edwardpeek" wrote:
I did actually used a global head model (7cm forward and 7cm up)

These seem too small. Most people are more like 12cm up and 12cm forwards (both plus or minus about 3cm std.dev) from pivot point to eyeball - the pivot point is roughly level with your jaw and below the back of your ear, though that's only a rough guide because the neck is a complex joint. DK2 and beyond fortunately make the head model less important - it's a difficult thing to measure.

"edwardpeek" wrote:
From my cursory review of the Oculus method, it appears you only use the head model during rendering, but not during warping, which I believe introduces a similar error.

Correct - we currently only support a rotation-based timewarp, because it's cheap and has few adverse effects, i.e. it never seems to make things look worse (though it might not make them look better). We are researching versions that also use position, but all the simple versions don't work well, and the complex ones chew a lot of rendering time. Still looking though.

I wonder where is the sweet-spot for frame rate up sampling.

Suppose you have a rendering engine which can dynamically adapt scene detail according to framerate requirements.
An application could leave out more detail to archive full frame rates 1:1 or render at 75Hz (or 90Hz).
But it might lead to a better experience to just render at some fraction like 1:2 like 37.5Hz (or 45Hz) and provide more detail and live with some artifacts from the time warp process.

"edwardpeek" wrote:
With the release of timewarping in the latest Windows 0.3.x SDK I figured I should dump a couple of paper's I've worked on regarding the technique. They're pretty theory-heavy, but there may be something of interest to other developers in there.

https://www.cs.auckland.ac.nz/~epee004/archive/WarpQuality.pdf
https://www.cs.auckland.ac.nz/~epee004/archive/HmdLatency.pdf

The methods discussed are slightly different to what's in the SDK but the core theory is the same. I'm happy to answer any questions people might have on the technique as well.

These are excellent papers! IHMO these kind of technologies will be crucial to overcome the performance bottleneck of high definitions combinded with low persistance (which relies on very high framerates)

spyro