compositor layer using for render 360 video make render result judder

[wangxiaogangan]

my team use UNDERLAY layer which shape = Equirect to render 360 video. the idea comes from oculus video app which looks like using UNDERLAY to make picture more sharp. platform is unity 2017.3.1 + oculus sdk 1.26

but i figure out OVROverlay.cs (oculus sdk 1.26) using 

private bool PopulateLayer(int mipLevels, bool isHdr, OVRPlugin.Sizei size, int sampleCount, int stage)  

which make gama correct  & copy texture into overlay SwapChain. because of blit operation kills performance, i do not need to render each frame until new frame come, i modify OVRoverlay's code looks like: 

void PopulateLayer()
{
    for (int eyeId = 0; eyeId < texturesPerStage; ++eyeId)
    {
        Texture et = layerTextures[eyeId].swapChain[stage];
        if (et == null)
          continue;
        
        // render for 360 top-bottom video
        //textures[0] is videotexture from video file
        if (eyeId == 0)
        Graphics.CopyTexture(textures[0], 0, 0,
                             0, textures[0].height / 2, textures[0].width, textures[0].height / 2,
                             et, 0, 0, 0, 0);
        else
        Graphics.CopyTexture(textures[0], 0, 0,
                             0, 0, textures[0].width, textures[0].height / 2,
                             et, 0, 0, 0, 0);
        
        ret = true;
     }
}

void Render()

{

    GetCurrentLayerDesc();

    CreateLayer();

    CreateLayerTextures();

    LatchLayerTextures();

    PopulateLayer();

    SubmitLayer();

}

void LateUpdate()

{

    if (newFrameCome)

        Render();

}

but when using underlay to play video, video texture  judders ( looks like media player play frame not in a queue) once in a few secs.

you can download the render result from https://www.dropbox.com/s/rhi4mjtwmj9ndn7/problem.mp4?dl=0. do not directly look at website, you need download and seek the video into 0:20, there are some dog running and they position looks like blink, the judder phenomenon occurs once in few secs.  but without underlay and using mesh to render, the video looks like right no judder.

the original video you can get from https://www.dropbox.com/s/tqdc9g8a48pkdph/normal.mp4?dl=0, you can seek into 0:07 and compare with the underlay version.

is anyone can explain what i do wrong with underlay? or give me  some theory suggestion for Equirect underlay?
(new frame come then do render is save a lot of performance. but if i make render in every frame, judder occurs still)

[wangxiaogangan]

update 7/22/2018:

the judder phenomenon looks like previous frame get rendered in wrong order, there is only one function to set overlay texture content is:

void PopulateLayer()

if i copy video texture into all layerTextures[eyeId].swapChain's texture, the judder disappear. but make fps only 30 for oculus go, because 3 copytexture in one frame. this is unsufferable. the modified code looks like

for (int eyeId = 0; eyeId < texturesPerStage; ++eyeId)
{
      for(int i = 0; i < stageCount; ++i)
      {
          Texture et = layerTextures[eyeId].swapChain;
          Graphics.CopyTexture()
      }
}

is the pre frame record in swapChain making video judder?

[wangxiaogangan]

my video info is : 4096 * 4096, video fps = 30 fps, video codec = hevc, top-bottom-3d

softaware = oculus video app & device = coulus go can play this video smoothly & sharply with underlay tech, i can confirm that my media player framework can play this video smoothly & device fps = 60fps also, but if i change render method from unity mesh to underlay, judder problem ocuurs. 

so, could you give me some suggestion for using Equirect underlay and avoid frame judder like the oculus video does?
 

[wangxiaogangan]

imperativity said:

You should have you application decode the video content directly to that texture to solve your judder problem.

my video texture render pipeline is :

1. new surfacetexture(  texture2d 1 new from native opengl)
2. copy surfacetexture into unity texture2d 2( new from unity texture2d ) after onframeavailable
3.copy unity texture2d 2 into layerTextures[eyeId].swapChain[stage];( we can call them all texture2d 3 )

could you explain more clearly for "directly to that texture"? which texture is "that texture" specify in my render pipeline ? 

[wangxiaogangan]

@imperativity thank you very much, overlay is more effective compared with underlay. when finish my work, i will post the 360 video render method in detail using overlay if someone interested

[cloud_canvas]

wangxiaogangan said:

@imperativity thank you very much, overlay is more effective compared with underlay. when finish my work, i will post the 360 video render method in detail using overlay if someone interested

Please do! You and I seem to be the Overlay Soldiers together at this point, would love to see what you're doing because it's somewhat relevant for me as well. Keep up the great hacking.