Forum Discussion
Idea - How to get fast head tracking with any frame rate
Here's an idea of how to get low latency head tracking with almost any variable frame rate.
Render the scene from the player's point of view with no distortion, but to a larger render target with a wider FOV. The increased FOV should be large enough to take into account how fast a user might turn their head in one frame (I figure ~8 degrees?);
Then at a faster update rate (60 or 120 times a second?) sample how much the HMD orientation has changed since the last full render and calculate how far you'd need to pan and rotate the image.
You can then sample from the render in the highlighted frame and run the distortion effect on this and send this to the Oculus's screen. This would would eliminate any variability in the update rate of the head tracking for the user and would not require games to run at a stable 60 fps to have a smooth experience. Note: The player's movement and the simulation of the world would still update at the normal frame rate.
Ideally this whole process could run in hardware on the Oculus Rift itself. Doing so would require the hardware be able to apply the distortion and to accept a render target larger than the screen itself, which would need to be aware of the orientation it was rendered to.
Alternatively, doing this on the GPU is quite tricky. Running parallel updates alongside the normal scene render is not possible in any GPUs that I'm aware of, it would require a separate render pipeline. One other application of this would be to simply apply this technique at the distortion step of the current frame. This would reduce the amount of latency in the update by a little less than 1 frame (~ 16ms-30ms).
Is my thinking correct here? Any see any reasons why this couldn't work?
Render the scene from the player's point of view with no distortion, but to a larger render target with a wider FOV. The increased FOV should be large enough to take into account how fast a user might turn their head in one frame (I figure ~8 degrees?);
Then at a faster update rate (60 or 120 times a second?) sample how much the HMD orientation has changed since the last full render and calculate how far you'd need to pan and rotate the image.
You can then sample from the render in the highlighted frame and run the distortion effect on this and send this to the Oculus's screen. This would would eliminate any variability in the update rate of the head tracking for the user and would not require games to run at a stable 60 fps to have a smooth experience. Note: The player's movement and the simulation of the world would still update at the normal frame rate.
Ideally this whole process could run in hardware on the Oculus Rift itself. Doing so would require the hardware be able to apply the distortion and to accept a render target larger than the screen itself, which would need to be aware of the orientation it was rendered to.
Alternatively, doing this on the GPU is quite tricky. Running parallel updates alongside the normal scene render is not possible in any GPUs that I'm aware of, it would require a separate render pipeline. One other application of this would be to simply apply this technique at the distortion step of the current frame. This would reduce the amount of latency in the update by a little less than 1 frame (~ 16ms-30ms).
Is my thinking correct here? Any see any reasons why this couldn't work?
22 Replies
- The only problem that it doesn't really work because you get perspective messed up even for a simple head rotation motion, not to mention for any linear motion, in-game or Rift's. It also cost too much.
Carmack proposes that it's possible to do a rendering update by shifting pixels depending on most recent readings and pixels' world coordinates. He also says that using simple black pixels for the pixels that "were" off-screen works surprisingly well and you don't need to render extra padding for this. - Check out the Time Warping section of John's Latency Mitigation Strategies Paper.
- Posted this on Reddit, posting here too. From the fact that this would work just as well for on-screen games I suspected this has already been done, and sure enough it has. See this paper and look up other works that reference it (this paper works out a lot of the kinks with your idea, including dealing with translation and occlusion):
Mark et al, Post-rendering 3D warping, 1997 Symposium on Interactive 3D Graphics
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13.1789&rep=rep1&type=pdf
Abstract: "A pair of rendered images and their Z-buffers contain almost all of the information necessary to re-render from nearby viewpoints. For the small changes in viewpoint that occur in a fraction of a second, this information is sufficient for high quality re-rendering with cost independent of scene complexity. Re-rendering from previously computed views allows an order-of-magnitude increase in apparent frame rate over that provided by conventional rendering alone. It can also compensate for system latency in local or remote display.
We use McMillan and Bishop’s image warping algorithm to re-render, allowing us to compensate for viewpoint translation as well as rotation. We avoid occlusion-related artifacts by warping two different reference images and compositing the results. This paper explains the basic design of our system and provides details of our reconstruction and multi-image compositing algorithms. We present our method for selecting reference image locations and the heuristic we use for any portions of the scene which happen to be occluded in both reference images. We also discuss properties of our technique which make it suitable for real-time implementation, and briefly describe our simpler real-time remote display system." - If the goal is to get smoother head tracking due to low fps (due to fill rate?) perhaps the size of the render target can be adjusted on the fly.
So use a 1.7x renderbuffer when barely moving your head for high quality, and switch to a 1x buffer for larger head movements (or maybe even lower than 1x). "Proton" wrote:
If the goal is to get smoother head tracking due to low fps (due to fill rate?) perhaps the size of the render target can be adjusted on the fly.
So use a 1.7x renderbuffer when barely moving your head for high quality, and switch to a 1x buffer for larger head movements (or maybe even lower than 1x).
I've been experimenting with this and it works pretty well. The tricky part is determining when to make the switch. The lower-res rending when turning your head isn't really noticeable. It would be noticeable for translations though.- When I first suggested doing this "trick" back in February, I also said we should add zoom support (PTZ = pan/tilt/zoom), in my "PTZ Tweening for low-power low-latency head-tracking" thread:
http://www.mtbs3d.com/phpbb/viewtopic.php?f=138&t=16543
There are a lot more details, related ideas, and suggestions in that thread, which is worth reading if you are interested in this stuff.
As I called it, the term "tweening" implies that the render frames are used as key frames, and "tween" frames are generated at high speed to morph or warp between key frames in the framebuffer. As described in my thread, you need a framebuffer larger than the visible image, so you have room to "pan and tilt" with the latest high speed tracker data. The key frames are rendered data, and the "tweened" frames are fast-morphed key frames using the latest tracker data (for lowest latency), as also described in this thread.
Interestingly, this concept was discussed (not using my name or terminology) in a recent video featuring Brendan Iribe talking about getting Rift latency under 15 msec by using faster OLED display technology. However, he did say that high-speed morphing of framebuffer content after 3D rendering, using the latest head tracker data, was essential to getting the latency down:
http://www.youtube.com/watch?v=tL6hVtEkV9Q&t=10m18s
EDIT: When an idea's time has come, multiple people will have that idea. - We've been playing with this in various forms. The reprojection is a little more complex than just a 2D pan of the image - it's a full rotation and projection onto a plane. But it works well for orientation, and will probably be included in the next SDK.
"tomf" wrote:
We've been playing with this in various forms. The reprojection is a little more complex than just a 2D pan of the image - it's a full rotation and projection onto a plane. But it works well for orientation, and will probably be included in the next SDK.
Well that's awesome.- My main purpose for doing this would be to render the VR environment at a lower speed (perhaps 15 to 24 FPS) while head tracking data can manipulate the rendered data to use the highest available video framerate (currently a maximum of 71Hz on the Rift DK, without color distortion). I am running my Rift at 71Hz framerate, and it is noticeably smoother.
Using a lower-speed VR environment render allows more detail to be maintained, especially on a low power device such as the Raspberry Pi.
Sadly, although the Ras Pi has hardware floating point, it is not supported by Unity3D because it uses an ARM6 processor (with FPU) while Unity supports nothing less than ARM7.
Anyway, decoupling fast head tracked video refresh from slower VR environment rendering like this will not only be worthwhile in the SDK, but perhaps essential on the Ras Pi, to allow "useful" levels of detail (which is why I started my "PTZ Tweening" thread.
Having things in the environment change slower than expected may give them a jerky "robot-like" motion, but as long as you can look around with smooth orientation updates it should be very immersive. Just realistic VR with "poorly maintained" jerky NPCs perhaps, which may add to the realism for robot NPCs, especially if they are grimy with leaking fluids painted on them.
;) - So, am I getting this right, in understanding that this correction will always take place, adjusting for motion between each frame's pre-render... err.. "camera time_instance position lock" (whatever that may really be called), and the time it is actually displayed, even should the HMD receive a bazillion frames every second?
(EDIT: I suppose it may be necessary to point out that the above was in response to what geekmaster said about Iribe's speech and tomf's catching on to it (...assuming that's what he did).
...And now that I actually clicked the link gm provided, I guess I did get it right; For every frame, also if it arrives on time: Mitigate latency by warping the rendered image to account for prediction errors, based on the latest sensor data, compared to the prediction the frame was rendered around -- heuristics on top of heuristics, one might say ;). I am also excited to hear, directly after, in the video, about not wasting processing power on rendering tons of ultimately unused frames in between screen refreshes -- never got why people seem to tie the entire simulation to the rendering that way, rather than doing it real-time (Elder Scrolls crazy physics, anyone?).
)
