Forum Discussion
Improve MovieTexture Playback on the Oculus (360 Video)
Hello Everyone,
I have a situation where I am trying to get a high resolution video wrapped in a 360 degree sphere designed for the Oculus Rift to playback in unity. So the scene is setup like so:
A sphere gameobject is in the center. The sphere material has a video mapped to it. I have tested this with 2 videos. 1 has a resolution of 4096x2048 and the other has a resolution of 3840x1920. The goal of this project is to have a 360 degree video player. Here is where the issues begin. In order to have a good framerate to provide the best possible experience in the rift, the video quality needs to be dropped tremendously to run at >75fps. But in order to feel immersed and have the highest quality I need to leave the resolutions the same. Right now with those resolutions my frame rate is about 5-10 fps. It is very jitter and laggy.
I have tried the following to help improve framerate:
Disable Vsync, Run in DX9, Use a lower poly sphere, Lower bit-rates when converting (lose quality), having the video read directly from the hard drive (SSD), having it read directly from unity's built in "resources" folder (http://docs.unity3d.com/ScriptReference/Resources.html), and go as far as to remove the audio and try only video since some people said that it was the audio that was making it lag, and I'm sure I'm forgetting others. Another thing I did was use the profiler since it is now free in Unity 5. The profiler kept saying 90%+ of my CPU Usage is going into a property called: Gfx.WaitForPresent. After doing a very extensive amount of research I learned more about it, but I'm still not 100% sure I understand it. The best explanation I found was (viewtopic.php?t=21833) and it is the post about halfway down by chrispruett » Wed Apr 08, 2015 2:40 am.
If it is worth mentioning, my unity version is: 5.0.1f1.
My question is, is there anyway to achieve a smooth video using MovieTexture? I have read that the OGG theora codec is not very good however I was able to keep the quality in the video after the conversion. It's just that MovieTexture is so resource hungry I'm not sure it is the best solution. Furthermore I have tried running this program on a very high end machine at work with 2 GTX Titan Z's SLI connected and it is still not going over 20 fps, if that. Is there any alternative to MovieTexture that I could use? I do not care if it is a free or paid asset.
One "exciting" post I can across someone mentioned that in Unity 5.1 they were going to update the OGG Codec to support 4k video and up to 3 1080p streams within the next 6 months and this was posted late January. Link can be found here and it is the very last 2 posts (viewtopic.php?t=18185)
I am open to any suggestions. I can provide more information and pictures if it will help.
Thank you for your time.
I have a situation where I am trying to get a high resolution video wrapped in a 360 degree sphere designed for the Oculus Rift to playback in unity. So the scene is setup like so:
A sphere gameobject is in the center. The sphere material has a video mapped to it. I have tested this with 2 videos. 1 has a resolution of 4096x2048 and the other has a resolution of 3840x1920. The goal of this project is to have a 360 degree video player. Here is where the issues begin. In order to have a good framerate to provide the best possible experience in the rift, the video quality needs to be dropped tremendously to run at >75fps. But in order to feel immersed and have the highest quality I need to leave the resolutions the same. Right now with those resolutions my frame rate is about 5-10 fps. It is very jitter and laggy.
I have tried the following to help improve framerate:
Disable Vsync, Run in DX9, Use a lower poly sphere, Lower bit-rates when converting (lose quality), having the video read directly from the hard drive (SSD), having it read directly from unity's built in "resources" folder (http://docs.unity3d.com/ScriptReference/Resources.html), and go as far as to remove the audio and try only video since some people said that it was the audio that was making it lag, and I'm sure I'm forgetting others. Another thing I did was use the profiler since it is now free in Unity 5. The profiler kept saying 90%+ of my CPU Usage is going into a property called: Gfx.WaitForPresent. After doing a very extensive amount of research I learned more about it, but I'm still not 100% sure I understand it. The best explanation I found was (viewtopic.php?t=21833) and it is the post about halfway down by chrispruett » Wed Apr 08, 2015 2:40 am.
If it is worth mentioning, my unity version is: 5.0.1f1.
My question is, is there anyway to achieve a smooth video using MovieTexture? I have read that the OGG theora codec is not very good however I was able to keep the quality in the video after the conversion. It's just that MovieTexture is so resource hungry I'm not sure it is the best solution. Furthermore I have tried running this program on a very high end machine at work with 2 GTX Titan Z's SLI connected and it is still not going over 20 fps, if that. Is there any alternative to MovieTexture that I could use? I do not care if it is a free or paid asset.
One "exciting" post I can across someone mentioned that in Unity 5.1 they were going to update the OGG Codec to support 4k video and up to 3 1080p streams within the next 6 months and this was posted late January. Link can be found here and it is the very last 2 posts (viewtopic.php?t=18185)
I am open to any suggestions. I can provide more information and pictures if it will help.
Thank you for your time.
9 Replies
- I don't think you will be able to squeeze the performance your looking for with 4K video and using Unity.
I developed MaxVR and 4K video was a huge challenge to get better smooth playback. It's a custom DirectX 11 and Directshow Engine however. Color conversion can be a large barrier in performance. Using OpenCL, AVX2 and SSE2 intrinsics with really good multithreading solved these performance issues. - I can confirm as well, I had the same issue in my own engine (playing theora ogg/ogv files only). after talking to another member on this forum I implemented dx11 multithreading for video and now I can throw any video size at the player it's great. previously I would get judder with the ovr sdk active but a separate thread fixed it, just wish I had looked into threading earlier in my engine development.
- Apologies for the late reply. Thank you both for your answers. I should definitely look into multi-threading. For now I have tested my project with a plugin called, "Unity AVProQuickTime" and I am able to play 4k 360 video with 75fps easily. Again, thank you both. I had been troubled with the issue for awhile.
- Glad you got it sorted. just curious about your system specs. my system isn't that powerful and i'm deving on a gtx650. I found I'm getting video judder (not hmd) @3840x1920. was wondering how that size runs on your system. I decided last week to switch to ffmpeg for video so I could play any format and am really happy with it, though was just wondering if system specs affect playback of video in engine or if there is something I can do other than multi threading to improve perfomance on 4k res I had previously only gone as high as 2880x1440 and that was ok to play but going higher I'm assuming I just need a better system (I also tested in riftmax ti seemed to be sluggish on that res also so am almost convinced this is the case). Also check out "Time Couch" in hd if you get a chance great little 360 flic, was great on the dk2 in my player.
- @genetransfer
Can you provide any information on how you implemented the dx11 multithreading to solve this problem. I can't seem to find any information on the forum.
My setup runs smooth until I play the video and then it drops to 55 fps, the OGV is only 1920 x 960 ARGB 32 Bit 1.2 MB.
My GPU is a GTX 780, and Intel i7-4770K CPU @ 3.5GHz, 16 GB RAM - so i doubt it is the hardware. "aps502" wrote:
@genetransfer
Can you provide any information on how you implemented the dx11 multithreading to solve this problem. I can't seem to find any information on the forum.
My setup runs smooth until I play the video and then it drops to 55 fps, the OGV is only 1920 x 960 ARGB 32 Bit 1.2 MB.
My GPU is a GTX 780, and Intel i7-4770K CPU @ 3.5GHz, 16 GB RAM - so i doubt it is the hardware.
I'll just have a coffee to wake up, but anything i can contribute would only be relevant if using a custom engine - as I'm not sure how unity or unreal would handle it.- For simplicity I built my video player into a dll to plug into my engine, I’ll probably release it for others to use after I release my player and have gotten some real world feedback.
When I play a video I launch a separate thread (_beginthreadex()-hunt down a tutorial on how to use) – I use #include <process.h> but you can use can use one of the other solutions if you feel more comfortable.
The directx11 side turned out to be not that complicated; after all I just needed to draw the texture.
First you want to check for multithreaded device (forgive ugly code – just to paint a picture)…D3D11_FEATURE_DATA_THREADING ThreadingCaps;
gD3D_Device-> CheckFeatureSupport(D3D11_FEATURE_THREADING,&ThreadingCaps,sizeof(D3D11_FEATURE_DATA_THREADING));
if (ThreadingCaps.DriverConcurrentCreates)
{
gHWMultiThreadingCheck = true;
}
else
{
gHWMultiThreadingCheck = false;
}
And then create a deferred context (this is in addition to you main context – don’t forget to release on exit).gD3D_Device->CreateDeferredContext(0, &gD3D_DeviceContext_Deferred);
now at the place you write to your texture – on the threaded side, map – unmap with the deferred context and create command list…gD3D_DeviceContext_Deferred->Map(pVideoPlayerData.RTT.Texture, 0, D3D11_MAP_WRITE_DISCARD, 0, &c->mappedResource);
/*buffer voodoo*/
gD3D_DeviceContext_Deferred->Unmap(pVideoPlayerData.RTT.Texture, 0);
SAFE_RELEASE(pVideoPlayerData.pCommandList);
gD3D_DeviceContext_Deferred->FinishCommandList(false,&pVideoPlayerData.pCommandList);
Now back in the main thread you execute your command list on the main thread with
your main context…if(pVideoPlayerData.pCommandList != NULL && gD3D_DeviceContext != NULL)
{
gD3D_DeviceContext->ExecuteCommandList(pVideoPlayerData.pCommandList, true);
SAFE_RELEASE(pVideoPlayerData.pCommandList);
gCanDrawVideo = true;
}
Those are just the highlights - if using unity or unreal, not sure if any of that will help as I haven't really explored those engines deeply. - Alas, thanks for the information and replying but I'm using Unity and what you are doing is way over my head! It is good that you clarified that for me though.
Although I'm no closer to working out why Movietexture results in judder experience with oculus.
Thanks A. - no problem, shame it couldn't help, it's hard when the tools are out of your hands.
I believe alot does depend on the codec and rate of play as well as size, but i think you should be getting better framerate as on my gtx650 I can get away with an youtube hd (1920x960)mp4 almost reaching 71-75 at all the times but there are headtracking skips occasionally as I move up from there now as I have switched my player to use libvlc instead of ffmpeg - this way I don't have to worry about codecs being installed on the system - but i think there might be a little more overhead than my ffmpeg implementation.
saying that the old ogg/ogv player i was using wasn't that efficient and didn't like playing large files, it was ok with the rez but larger video just massacred it, so I switched to ffmpeg - but the build i was using had a few ugly issues so ended up going with vlc - oh so beautiful to work with :). I suppose you might be able to try a different video player plugin as mentioned by vectorizing in this thread or ask him what success he has had with it to see if it suits your needs - though i think the unity video player plugins are expensive not too sure though.
wish you the best in getting it sorted, though it probably just depends how the video player was intended to be used.
