Skip to main content
Up and Coming
January 16, 2014

Low persistence with (nearly) 60 fps performance possible?

Hi there,

As we know, low persistencedoes not work with 60 fps. As the screen flashes only once per frame and stays black until the next frames arrives, the flickering would be very annoying at 60 Hz. So we need 90-120 Hz to avoid this. If we just would use the same frame twice, we would just replace motion blur with heavy ghosting (the old frame could be seen twice on different places as we move our head).

So do we really need a higher framerate (and those way much processing power and bandwidth) just to solve this? Maybe there is another solution. Shutter glasses (as they are used for 3D-TVs) refresh side only with 60 Hz in alternating order. As the user sees through both sides at once, the two frequencies 'add' to 120 Hz for the visual system (which seem to be the important point here).

So the basic idea is, that we could just interleave the left and right perspective of TWO CONSECUTIVE frames in a way we get a "fake-120 fps"-Signal to the Rift:

First, the LEFT PERSPECTIVE OF FRAME 1 will be scanned out to the display (the right perspective of this frame will NOT BE RENDERED AT ALL):



After about 1 ms, the display turns black (either the backlight- or OLED-Switch):



After another 7 ms, the RIGHT PERSPECTIVE OF FRAME 2 arrives based on new data (the left perspective of frame 2 will not be rendered either):



After another black phase of 7 ms, the process repeats.

Again, it's important to note that the left and right perspective are not from the same frame but from different points in time, so the right perspective is NEWER than the left one (otherwise we would produce heavy ghosting as well).

Here is simple diagram I made in a few minutes to show the principle:



Here is a short testvideo which could be watched on a Rift: https://dl.dropboxusercontent.com/u/24358204/oculus/interleaving/oculus_interleaved_60Hz.mkv

If you stop the video on any frame you will notice that only one perspective is visible at once. As the Rift-DK1 only supports 60 Hz, you will easily notice heavy flickering, of course. But with a 120 Hz panel (and 120 fps 'perspective alternating rendering') the flicker should disappear.

The Rift uses lenses to unwarp the horizontally compressed images in a way they fill out nearly the whole FOV, so that's basically the same effect as with shutter glasses. Each side for it's own flickers at 60 Hz (with an offset of 8 ms), but they add to 120 Hz because both sides are watched 'over' each other and note side-by-side.

Note: The brightness of the picture will degrade by 50 % (because only half of the photons hit each eye in given time now). But that shouldn't a problem in a closed box directly in front of your eyes so I assume this can be callibrated.

The big advantage is, that you will have to render the same amount of pixels like on 60 fps. So this save a considerable amount of GPU time and memory bandwidth when compared with "progressive" 120 fps rendering. As the CPU has to do all simulation and world processing 120 times/sec. no matter what, it's only a solution on games which are rather GPU- than CPU-limited (which should be the case on high resolutions and frame rates).

What do you think? Could this actually work? Without a DK2 and a low persistence display it's hard to tell but maybe there is a fundamental flaw in this approach?

spyro

    16 replies

    spyroAuthor
    Up and Coming
    January 20, 2014
    "jherico" wrote:
    According to the slides from the talk Michael Abrash gave at steam dev days, the low persistence display should be limited to about 3ms of being on. Because of the short duration, you need a high refresh rate to avoid flicker, which they recommend at at least 95 hz. If you had to do that per-eye it would result in a total 190 hz, which seems... unlikely.


    Sounds very reasonable to me. But it's important to understand that this frequency does not stand for 'each eye' but for the needs of the visual system itself. I was hoping, that this could work because the 2x60Hz in alternating order (with each perspective 'up to date') would 'fuse' to 120 Hz because of the highly overlapping FOV of both eyes.

    IHMO it's good news that we will 'only' need 95 fps instead of 120. However, when you assume a 2560x1600 panel in the CV1 @95Hz, rendering a modern game will easily kill every system except the really high-end rigs out there. According to the Oculus SDK you will have to render another 40% more pixels (see http://static.oculus.com/sdk-downloads/documents/OculusBestPractices.pdf, pg. 14) because of the barrel distortion.

    Maybe, some tricks like Z-Buffer stereo reprojection (like VorpX) could be the solution to save performance. I'm afraid, with brute force there is no way to pump all this pixels out of modern game engines...

    Another solution could be something like Dmitry Andreev's 'Realtime framerate upscaling' (based on the current frontbuffer and the Z-Buffer of the frame which the GPU is working on next): http://and.intercon.ru/rtfrucvg_html_slides/

    spyro
    Inspiring
    January 20, 2014
    "spyro" wrote:
    IHMO it's good news that we will 'only' need 95 fps instead of 120.


    You still need to render two full images in that space of time (a little over 10ms) to

    However, when you assume a 2560x1600 panel in the CV1 @95Hz, rendering a modern game will easily kill every system except the really high-end versions.


    I think as we move into actual consumer products, we're going to see a lot more retro / stylized applications that shy away from the super complex lighting and find detail that's common in today's AAA games, going more for cell shading, or tron-esque style visual design, sacrificing photo-realism for presence.

    According to the Oculus SDK you will have to render another 40% more pixels (see http://static.oculus.com/sdk-downloads/documents/OculusBestPractices.pdf, pg. 14) because of the barell distortion


    Well, I'm not sure about that. There's actually a significant amount of pixels (and thus time) that could be dropped from the rendering because it's out of the FOV of the user. The current samples don't demonstrate this, and to do it properly you'd have to find the inverse distortion. But once you've got that, you'd be able to create a rendering stencil that would drop any pixels that were sufficiently far from the lens center. Basically areas towards the corners of the rendered image. It won't shave off a lot, but it won't be insignificant either.

    Maybe, some tricks like Z-Buffer stereo reprojection (like VorpX) combined could be the solution to save performance. I'm afraid, with brute force there is no way to pump all this pixels out of modern game engines...


    The number of pixels isn't the problem (well, not all of it). Modern GPUs can easily draw that many pixels. It's the per pixel work being done, and not in the distortion either, but in the scene rendering. Doing the distortion effect on a 960x1080 scene takes on the order of 10s of microseconds on my desktop. Even if you scale that up by a factor of four, you're still not talking about a significant amount of the whole rendering budget.
    Brad Davis - Developer for High Fidelity Co-author of Oculus Rift in Action
    Up and Coming
    January 20, 2014
    "spyro" wrote:
    "halopend" wrote:
    Nah. The point of low persistence is your eyes only see something for a short bit WHILE ITS CORRECT.

    With your setup either one eye is always "lagging" more than the other


    Actually that should not be the case, because the two perspectives are NOT from the same point in time as I pointed out several times in my post. Every frame is based on the most current simulation state. It is actually a 120 fps rendering with 120 Hz but in each frame you render only one perspective, essentially 2D. Your brain would have to take the depth information from the last frame.

    Of course, real 120 fps are better. That is not the point. There is not more visual information in the signal compared with 60 fps. But this could probably be a way to avoid the flickering when combined with low persistence.

    Maybe this doesn't work, but then there must be another reason for that.


    Yeah, sorry. I just looked at the pretty pictures and probably only read the title of the thread!

    It might not of been clear, but I was actually saying what you wrote!! That testing out 120hz (60 per eye) is worth trying out. It has more overhead than 60Hz rendering both eyes simultaneously, but if it tricks the eyes into believing what its seeing its worth testing!
    Inspiring
    January 22, 2014
    I watched the video. I experienced a lot of flicker.

    From what I understand, you need greater than 60Hz for low persistence to look good.

    Also, there doesn't seem to be any gain from alternating eyes. Why not just render both?
    AMD Ryzen 7 1800X | MSI X370 Titanium | G.Skill 16GB DDR4 3200 | EVGA SuperNOVA 1000 | Corsair Hydro H110i Gigabyte RX Vega 64 x2 | Samsung 960 Evo M.2 500GB | Seagate FireCuda SSHD 2TB | Phanteks ENTHOO EVOLV
    spyroAuthor
    Up and Coming
    January 23, 2014
    "cybereality" wrote:
    I watched the video. I experienced a lot of flicker.
    From what I understand, you need greater than 60Hz for low persistence to look good.


    That's why I wrote THIS directly under the URL to the video:

    "As the Rift-DK1 only supports 60 Hz, you will easily notice heavy flickering, of course. But with a 120 Hz panel (and 120 fps 'perspective alternating rendering') the flicker should disappear."

    "cybereality" wrote:

    Also, there doesn't seem to be any gain from alternating eyes. Why not just render both?


    And again:

    "The big advantage is, that you will have to render the same amount of pixels like on 60 fps. So this save a considerable amount of GPU time and memory bandwidth when compared with "progressive" 120 fps rendering. As the CPU has to do all simulation and world processing 120 times/sec. no matter what, it's only a solution on games which are rather GPU- than CPU-limited (which should be the case on high resolutions and frame rates)."

    It seems that nearly nobody had the time to read the original posting first... ;)

    The basic idea is, that the GPU renders the same amount of "perspectives" and pixels as at 60 fps. My hope was, that the brain could 'fuse' the two alternating 60 Hz Signals for the left and right eye to 120 Hz as it overlays the two pictures from both eyes. The two perspectives are not from the same time frame but were calculated from the most current world state. My test video does this with only 2 x 30Hz to show the basic idea on a 60 Hz panel.

    To save additonal power, every 2nd perspective could be calculated just with an updated projection matrix instead of a 'real' new calculated world state. So every character in the world would be on the same place as before, but more than 60 fps are AFAIK only really useful if you look around with your head so maybe it doesn't matter at all.

    Maybe I should have waited for the DK2 with this. Maybe it simply doesn't work because of flickering in the peripheral view where the visual system get's only pixels from one of the two perspectives.

    spyro
    Up and Coming
    January 24, 2014
    Haha, it looks like great minds think alike. I had a similar idea a couple of days ago with CastAR, except in addition to temporal shifting per-eye, there is also temporal shifting per-color (since CastAR's FLCoS projectors display color sequentially). As your eye is in smooth pursuit of a moving object/scene, the three offset colors will land on the same place on the retina, producing an image with high motion fidelity and without color fringing or ghosting.

    It makes a bit more sense in CastAR since the eyes AND colors are already temporally offset. So the changes needed are purely on the software side (special shaders that temporally offset colors/eyes relative to camera and object movement; similar to velocity-vector based motion blur shaders).

    I made a thread about it on the technical illusions forums: http://technicalillusions.com/forums/thread-401.html

    P.S. If anyone has an ST1080, PLEASE TRY THE DEMO AT THE BACK OF THE THREAD AND REPORT BACK!

    P.P.S. I don't mean to steal any of your thunder, spyro. It's a great idea.

    EDIT: hmmm, upon closer inspection, it appears our ideas aren't as similar as I'd first thought. The great benefit of yours is to reduce GPU load? Perhaps it might be an acceptable trick on some of the more hardcore games, or for consoles...

    There was a pair of glasses a while back that Nike touted to improve depth perception. They were like shutter glasses. I think they were called the Nike Vapor Sparq or something. Perhaps you'll end up with a similar effect.