Forum Discussion
Solved
How to handle scene loading in Unity like when using SteamVR (compositor)
I'm currently evaluating how feasible adding a native Oculus SDK version of Holodance is and the greatest challenge I see so far is porting our approach to loading scenes over.
The way we currently do this is based on Valve's SteamVR_LoadLevel example. In a nutshell: Set the skybox for the compositor, set a load texture in front of the player in the compositor, fade the game to the compositor, once that is done, stop rendering to VR (because the player now only sees what's in the compositor), start loading the scene and wait until it's done, do other things that could cause hiccups like prewarming shaders and a garbage collection round, start rendering to VR, fade back in. Play.
In addition to what the standard SteamVR_LoadLevel does, we render a preview of the level that is being loaded, with some text and a progress bar using Unity UI, into a RenderTexture that we also show and update on changes in the compositor. Also, to make things a little more fancy, we have several textures on quads to have our logo in 3D on several planes. Theoretically we could even animate those planes but Unity does have some hiccups while loading levels (even when using async and additive), so we don't do this.
Reviewing the API provided by Oculus Utilities for Unity 5.x, it seems that one class we could use is OVROverlay. Putting the texture to the right position is simply by using the Transform location and mesh primitive, right?
Then, there's OVRScreenFade, which lets us fade ... but only to a color.
So ... it seems there are two things missing: How can we fade full to the compositor and stop rendering to VR while we are doing things what would cause framedrops?
When using SteamVR / OpenVR, this is done by calling CVRCompositor.SuspendRendering(true).
The way we currently do this is based on Valve's SteamVR_LoadLevel example. In a nutshell: Set the skybox for the compositor, set a load texture in front of the player in the compositor, fade the game to the compositor, once that is done, stop rendering to VR (because the player now only sees what's in the compositor), start loading the scene and wait until it's done, do other things that could cause hiccups like prewarming shaders and a garbage collection round, start rendering to VR, fade back in. Play.
In addition to what the standard SteamVR_LoadLevel does, we render a preview of the level that is being loaded, with some text and a progress bar using Unity UI, into a RenderTexture that we also show and update on changes in the compositor. Also, to make things a little more fancy, we have several textures on quads to have our logo in 3D on several planes. Theoretically we could even animate those planes but Unity does have some hiccups while loading levels (even when using async and additive), so we don't do this.
Reviewing the API provided by Oculus Utilities for Unity 5.x, it seems that one class we could use is OVROverlay. Putting the texture to the right position is simply by using the Transform location and mesh primitive, right?
Then, there's OVRScreenFade, which lets us fade ... but only to a color.
So ... it seems there are two things missing: How can we fade full to the compositor and stop rendering to VR while we are doing things what would cause framedrops?
When using SteamVR / OpenVR, this is done by calling CVRCompositor.SuspendRendering(true).
- Hey Jashan
Thanks for the explain.
For this kind of feature, you actually can use OvrOverlay to do that.
for example
1. you can clear your game background to black
2. make sure ovrOverlay ( compositor layer ) is rendered ( at least send into compositor once )
3 then enabling your CPU heavy work.
4. you should be able to see ovrOverlay updating with correct tracking even you background app doesn't update. Technical speaking, ovrOverlay ( compositor layer ) was send into the async timewarp, will update there all the time with 90 FPS regardless your game's FPS.
I understand this way might be not friendly enough for most developers, we are considering to provide a more specific prefab / or function to do this automatically for developers, however, so far, I can't promise the specific time frame for this feature
jashan said:
... also, does anyone know why sometimes, the compositor does no head-tracking (i.e. all layers from the compositor "move with the head", or stuck in front of your face)?
Apparently, no one knew. I don't know for sure - but here's what I found out so far. This issue put me near giving up, so I'm leaving this here in case someone else runs into the same issues; this might save you some headaches (and Oculus engineers, please at least read the last paragraph):
It had something to do with the scene: I could grab the file "level0" from one build where it worked correctly, and use that to overwrite that same file in a build where it didn't work, and that way I could "fix" such a broken build (and also, break a build where it worked by doing it in the opposite direction). Unfortunately, I'm not aware of a decent way to look for differences in two such files (they were also the exact same file size). Mind you: Those are not the scene assets in the Unity projects, but the binary levelN files in the build.
I did have sessions where I could reproduce it in the Unity editor for a while, but even when Unity said the current scene had no changes compared to what's stored on the disk, making a copy of that scene, and switching over to that copy would make it work in the editor again. Then, switching back to the scene where it didn't work just a minute ago would keep it working. It was not easy to get the Unity editor into the state of this not working, and I don't know for sure what exactly caused it - all I know is that it usually happened after doing a build that produced a faulty build.
In one of the sessions where it didn't work in the editor, I spent quite some time trying to debug OVROverlay to figure out what might be causing this. The most likely candidate obviously was the headLocked flag, which gets set when the OVROverlay is attached to a descendant of the main camera. I could not validate this to be the cause of the issue - headLocked was never set to true in any of my test-sessions but they did show the faulty behavior. One possible explanation could be that this somehow got cached in the runtime handling the compositor from a previous session, but that's just random guessing.
Still, not knowing of a better way to solve this, I removed this whole headLocked stuff (no responsible VR developer should use headlocked stuff, anyways, so it's not like I lost anything of value). Also, as I saw that OVROverlay uses Camera.main, and you probably use that more often, I made sure to only have the VR camera tagged with MainCamera.
From the looks of it, this did fix it; potentially due to Camera.main behaving more as you expect it in some other places of your integration. The thing is: Camera.main is not a reliable way to check for VR cameras. If you check for Camera-components where Camera.stereoTargetEye is not StereoTargetEyeMask.None, you actually reliably do what you probably thought you were doing by using Camera.main.
18 Replies
Replies have been turned off for this discussion
imperativity said:
Hi,
Thanks for this suggestion. We are road-mapping this for future implementation and looking into providing a workaround in the meantime for your use. I will update you when I do!
Great, thank you!- Hey Jashan
Want to understand you request better, are you asking the ability to display compositor layer when CPU is dedicating loading your scene ( no scene rendering ) ?
Jian.
JianZ said:
Want to understand you request better, are you asking the ability to display compositor layer when CPU is dedicating loading your scene ( no scene rendering ) ?
Hi Jian,
yes, that's basically what I'm looking for. But actually, what I'm looking for is a lot more specific than that:
Loading a new scene is the primary use case because even when loading scenes additively and asynchronously, Unity will create rather severe hiccups. But there are other use cases as well, like preparing object pools or triggering garbage collection at a well-defined (and non-critical) point in time. Unity can even give you a fairly intense framedrop when activating an object in the scene for the first time, so that's another thing I've started doing in this process. Another situation where I may use this is loading and decompressing MP3s from the file system. While using separate threads helps a lot, there are still operations that cannot be done outside the game's main thread that can take 50, 100 or even more ms, which is far from acceptable in a VR game.
Many of those operations are batched with scene loading because fading out and back in takes a little time, so I don't want to do this too often. But loading MP3s is one example that is completely unrelated to scene loading.
Basically, what I'm looking for is a controlled way to activate / deactivate the compositor from within the game, at any time, to make sure situations where the game does cause hiccups that cannot be prevented will not cause discomfort for players.
Since Unity added the terrible "enable VR for your game by ticking a single checkbox feature" (a design that broke a lot of flexibility concerning cameras and tracking for no good reason), I'm very cautious about anything that sounds like an assumption for a specific use case that would make the solution useless for a lot of other use cases. Like, I wouldn't be surprised if Unity's answer to what I'm looking for would be activating the compositor while loading a scene, which would be a terrible idea IMHO. It would, of course, be nice for Unity users that don't know what they are doing (like that "enable VR with ticking a checkbox"-monstrosity), but for professionals, that kind of approach is a slap in the face.
The solution that SteamVR offers gives me full control and I can even design this in a way that is visually entertaining and gives players fairly good feedback on progress (scene loading in Unity unfortunately has progress apparently stand still for a little while, then make a big jump when loading is almost complete, so it's far from perfect but still okay enough).
Sunny regards,
Jashan- Hey Jashan
Thanks for the explain.
For this kind of feature, you actually can use OvrOverlay to do that.
for example
1. you can clear your game background to black
2. make sure ovrOverlay ( compositor layer ) is rendered ( at least send into compositor once )
3 then enabling your CPU heavy work.
4. you should be able to see ovrOverlay updating with correct tracking even you background app doesn't update. Technical speaking, ovrOverlay ( compositor layer ) was send into the async timewarp, will update there all the time with 90 FPS regardless your game's FPS.
I understand this way might be not friendly enough for most developers, we are considering to provide a more specific prefab / or function to do this automatically for developers, however, so far, I can't promise the specific time frame for this featureHello JianZ,
Thank you for your solution.
I have used the OVROVerlaySample and I have replaced SimulateLevelLoad(); with scene loading, but the loading message is not displayed and the scene is frozen until the next scene is loaded.
here is the code:
IEnumerator WaitforOVROverlay() { Transform camTransform = mainCamera.transform; Transform uiTextOverlayTrasnform = loadingTextQuadOverlay.transform; Vector3 newPos = camTransform.position + camTransform.forward * distanceFromCamToLoadText; newPos.y = camTransform.position.y; uiTextOverlayTrasnform.position = newPos; cubemapOverlay.enabled = true; loadingTextQuadOverlay.enabled = true; noneRadioButton.isOn = true; yield return new WaitForSeconds(0.1f); //Here is my scene loading SceneManagerWithParameters.Load(sceneMaison, "", ""); cubemapOverlay.enabled = false; loadingTextQuadOverlay.enabled = false; yield return null; }Is there something wrong ?
I got it working using Async Loading for the scene:
void TriggerLoad() { StartCoroutine(WaitforOVROverlay()); StartCoroutine(LoadYourAsyncScene()); }IEnumerator WaitforOVROverlay2() { Transform camTransform = mainCamera.transform; Transform uiTextOverlayTrasnform = loadingTextQuadOverlay.transform; Vector3 newPos = camTransform.position + camTransform.forward * distanceFromCamToLoadText; newPos.y = camTransform.position.y; uiTextOverlayTrasnform.position = newPos; cubemapOverlay.enabled = true; loadingTextQuadOverlay.enabled = true; noneRadioButton.isOn = true; yield return new WaitForSeconds(1.0f); yield return null; }IEnumerator LoadYourAsyncScene() { // The Application loads the Scene in the background as the current Scene runs. // This is particularly good for creating loading screens. // You could also load the Scene by using sceneBuildIndex. In this case Scene2 has // a sceneBuildIndex of 1 as shown in Build Settings. AsyncOperation asyncLoad = SceneManager.LoadSceneAsync(myScene); // Wait until the asynchronous scene fully loads while (!asyncLoad.isDone) { yield return null; } }
JianZ said:
Hey Jashan
Thanks for the explain.
For this kind of feature, you actually can use OvrOverlay to do that.
for example
1. you can clear your game background to black
2. make sure ovrOverlay ( compositor layer ) is rendered ( at least send into compositor once )
3 then enabling your CPU heavy work.
4. you should be able to see ovrOverlay updating with correct tracking even you background app doesn't update. Technical speaking, ovrOverlay ( compositor layer ) was send into the async timewarp, will update there all the time with 90 FPS regardless your game's FPS.
I understand this way might be not friendly enough for most developers, we are considering to provide a more specific prefab / or function to do this automatically for developers, however, so far, I can't promise the specific time frame for this feature
Sorry for getting back to this after a little delay ... but ... this looks like what I was looking for. I don't mind a more "barebones" way of accessing this - actually this is usually preferable over something that gives me less control.
Aside of giving it a try and seeing how this works in practice, I just have one more question: Do you have a skybox feature built-in? Or would you simply create 6 quads representing a huge cube at a very large distance (well, obviously, the cube "walls" are at a large distance, you'd be in the middle of the cube)?- yes， the ovroverly support cubemap, you can select it from overlay shape type, ( a bit of more implementation detail, it is implemented natively on mobile，but faked by method like what you said on PC since sdk feature differences, you shouldn't see big difference as a end user)
- So, this has been a while and I'm now (finally) trying to integrate this.
The main problem I'm running into is that OVRScreenFade apparently breaks the overlays (one of the overlays, to be specific). I have all my overlays set to "No Depth Buffer Testing", so IMHO, they should always be rendered over anything coming from Unity but when I use OVRScreenFade, my foremost overlay at first does not render at all, and then only renders the top half. In fact, it looks like only 3 overlays are supported (instead of 15). This is in Unity 2017.4.
Also, I'd really like to stop rendering to the compositor while loading, like I can with SteamVR/OpenVR (see above - CVRCompositor.SuspendRendering(true)). For now, I just disable the VR camera once I switched to the overlay-based loading screen.
One really unfortunate thing that happens only in builds: The overlays are stuck to the camera. In the editor, the behavior is correct, there, the overlays are located in the world.
Finally, is there a good way to fade the overlays in and out? Having them pop in and out is not really a nice user experience. - that's strange actually, OVRScreenFade is a MonoBehaviour class, it only fade the eye buffer, compositor layer was often called Overlay, so it isn't part of the eye buffer, it is rendered independently in the compositor. it is unlikely eye buffer content to affect compositor layer result, guess something else caused it, if you can use renderdoc to do a capture , you should be able to figure out why.
for your last question, I believe you use OvrManager.SetColorScaleAndOffset to controll overlay's blending - Thank you, Jianz! OvrManager.SetColorScaleAndOffset(...) does help a lot.
Am I right to assume that there is no way to control compositor/overlay transparency separately from the eye buffer (I'm assuming "eye buffer" is what's rendered by the game)? Or a way to fade individual compositor overlays in/out?
Assuming that I can only control the composited image (compositor/overlay + eye buffer coming from the game), for now, what I do is use OVRManager.SetColorScaleAndOffset(...) to fade everything to black, then activate the overlays and compositor skybox, then fade back in, and when switching back, again, fade to black, switch off overlays and compositor skybox, fade back in. With SteamVR, I can do a crossfade between the compositor and what the game renders but fade to black and back will do.
One question, though: If SetColorScaleAndOffset has an effect on everything, what is the purpose of being able to change alpha (i.e. why a Vector4 instead of Vector3)?
[EDIT: Oh ... actually, it turns out that SetColorScaleAndOffset apparently does not effect the Overlay cubemap, so it seems I cannot fade the "compositor cubemap" in/out this way. Am I missing something or is this really not possible?]
I'm not sure about the purpose of OVRScreenFade: The results I've had with this were very mixed, with parts of the old image still being visible when everything should already have been faded to black. In my game, it was very obvious that this basically just puts a black plane in front of the camera. Using SetColorScaleAndOffset seems to work much better and more reliably. Maybe OVRScreenFade should also use that instead?
I'm sorry I can't be more helpful with the issue with OVRScreenFade vs. OVROverlay. All I can say is that when I used OVRScreenFade, OVROverlay behaved very unreliably, or actually, to be precise, it reliably produced unexpected results (i.e. every overlay after the third not showing up, sometimes the fourth overlay showing up split in half, sometimes those overlays appearing when looking down and then back up). Now that I have removed OVRScreenFade, OVROverlay behaves well. For me, the issue is solved because it turns out I don't need OVRScreenFade at all ;-) - Hey @jashan,
Good question. When you're trying to perform any type of fade, SetColorScaleAndOffset() is the way to go. It's a simple formula, where in the last step of the compositor, the color value is multiplied by colorScale and added to colorOffset, and these values can be changed per-frame.
Currently, we've exposed 2 mechanisms for color scale and offset. One which sets a given color scale and offset for all layers, and one which sets these just for the eyeFov layer. The API was designed like that mainly because colorScale is often a global property that developers want to apply to all layers, and an obvious use case is a fade-to-black, for which the colorScale starts at (1, 1, 1, 1) and ends up at (0, 0, 0, 0). Sometimes for splash screens, you might also only want to change values for the eyeFov layer.
As regards per-layer color scale and offset, we are adding it very soon, hopefully in our next release. Right now you can only apply global or only eyeFov color scale as outlined above, but soon, you'll be able to set it per-layer, so for your case, if you have the cubemap that you want to fade in/out, and affect nothing else in the scene, you'll be able to do that soon.
