What optimizations can be implemented for realtime specular on mobile VR?

[firagabird]

I'm heavily invested in implementing stereo-correct specular highlighting in Gear VR. One of my biggest issues with almost all existing Gear VR titles is that the best looking ones still have a dull, unrealistic look. My working theory which I'm looking to test is that it has to do with the complete lack of specular highlighting.

Of course, a full featured reflection shader will kill mobile performance, so that's of the table. Normally, the right optimization would be to use specular & roughness maps (Daedalus for Gear VR uses this). Unfortunately, this is still incredibly GPU intensive, and causes heavy aliasing if done wrong.

The suggestion proposed by John Carmack would be to do specular highlighting in realtime, but in the simplest possible case: one directional light e.g. the Sun, the camera & normal angles calculated in the vertex shader, and the light angle & specular highlight itself calculated in the fragment shader.

Even with this simple case, fairly advanced functions (pow, cos) need to run on the vertex & fragment shaders. I was wondering: has anyone tried using LUTs (look up textures) to speed up the calculations? Is the bandwidth tradeoff acceptable in a real world Gear VR game? What other optimizations could I try to speed up the process of calculating specular highlighting in realtime?

[Norman3D]

Hi!
Norman, Technical Artist for Daedalus here. Just wanted to address really quickly the use of specular in Daedalus. We have two different material quality levels for Daedalus and depending on the device we use one or another. These quality settings turn on and off certain graphical features.
We are not exactly sure what device Carmack used to play Daedalus. He mentioned on his twitter that he used OculusGo. Most likely the high quality material level was being used on that device, despite it having lower specs than an S8, so that might have caused the framerate drops he mentioned. Another possibility is that he was using the american model of the S7, which is significantly slower than the international model of the S7 when running OpenGL ES 3.1 (which we need to have pre-filtered cubemaps). We are a very small studio based in Spain with little to no resources, so getting our hands on an american S7 for profiling was not possible at the time. We had to be rather conservative with our optimization for S7 due to this since we were completely blind as to how it would perform on the american model. So it might be possible that we are not running at 60fps all the time on that specific device. Oculus analytics however tell us that we are maintaining 60fps most of the time without a problem.
I encourage everyone to make use of cubemaps but to optimize them as much as possible wherever you can.
There are few different options in terms of scalability for reflections that you can make use of, we made use of almost all of these combinations for Daedalus.
- No Cubemap
- Cubemap with uniform to control roughness 
- Cubemap with texture to control roughness
- Cubemap with uniform to control roughness + parallax correction
- Cubemap with texture to control roughness + parallax correction
- Cubemap with uniform to control roughness + normal map
- Cubemap with texture to control roughness + normal map
- Cubemap with uniform to control roughness + parallax correction + normal map
- Cubemap with texture to control roughness + parallax correction + normal map

The game runs on S6 for 45+ minutes, and on the S7 it should last way longer, we haven't even tested it as our min requirement was S6 for 45+ minutes. A lot of these optimizations are necessary not to maintain 60fps at GPU 2, but to keep temperature low enough for the S6 to not hit thermal limit.

Fun fact: We had all materials with parallax corrected cubemaps running on the international S7 for 45 minutes at 60fps on Android 6 without a problem. Unfortunately Android 7 seemed to add a performance overhead on the driver level so we could not maintain the 60 fps, so we were forced to use the lower quality level for those devices.

[firagabird]

@Norman3D 

Thanks for the highly detailed insight into your game's graphics implementation.

Your wholehearted recommendation for using cubemaps is incredibly surprising, and seems in contrast with Oculus & Carmack's guidelines. It really goes to show that very few rules in VR are a given. Beyond specific implementations (which I'll definitely try!) I think the biggest takeaway for graphics is: don't knock it till you try it.

Re: Oculus Go, it actually uses a Snapdragon processor (SD is also used in American S7's) which would explain the similar performance profile between the two. It's really unfortunate that Exynos processors behave differently, and that some things that run well on them are less performant on their Snapdragon equivalents. I was kind of hoping for Oculus Go to be an undisputedly faster device than at least both S7 models.

[Norman3D]

firagabird said:

@Norman3D 

Thanks for the highly detailed insight into your game's graphics implementation.

Your wholehearted recommendation for using cubemaps is incredibly surprising, and seems in contrast with Oculus & Carmack's guidelines. It really goes to show that very few rules in VR are a given. Beyond specific implementations (which I'll definitely try!) I think the biggest takeaway for graphics is: don't knock it till you try it.

Re: Oculus Go, it actually uses a Snapdragon processor (SD is also used in American S7's) which would explain the similar performance profile between the two. It's really unfortunate that Exynos processors behave differently, and that some things that run well on them are less performant on their Snapdragon equivalents. I was kind of hoping for Oculus Go to be an undisputedly faster device than at least both S7 models.

Hardware specs for the Oculus Go are not confirmed afaik. Even if it's the same chip as american S7 (Snapdragon), you can't forget that multires will be implemented at the driver level, giving us at the very least 15% more pixel shader, depending on your foveation level.
My point with the post above is that cubemaps can be used, just make sure to add them where it counts. All the materials in Daedalus for the S8 have parallax corrected cubemaps, with roughness maps and normal maps. On the S6 the concrete surfaces have no cubemap at all, and only the metallic surfaces have a cubemap with roughness map, but no normalmap and no parallax correction.
If your game features no locomotion then parallax corrected cubemaps make little sense for example, as the parallax can only really be perceived when moving the camera around.
It's not all black or white, there are definitely shades of grey when it comes to adding graphical features to your gear vr games.

[firagabird]

Norman3D said:

Hardware specs for the Oculus Go are not confirmed afaik. Even if it's the same chip as american S7 (Snapdragon), you can't forget that multires will be implemented at the driver level, giving us at the very least 15% more pixel shader, depending on your foveation level.

You‘re right, multires is gonna be a big deal for chlus Go. Hopefully the gains (plus other standalone optimizations & maybe a stronger chip) are enough to at least match the perf of an Exynos S7.

My point with the post above is that cubemaps can be used, just make sure to add them where it counts. All the materials in Daedalus for the S8 have parallax corrected cubemaps, with roughness maps and normal maps. On the S6 the concrete surfaces have no cubemap at all, and only the metallic surfaces have a cubemap with roughness map, but no normalmap and no parallax correction.

The different variations of material quality across devices is really fascinating to think about. Is there any chance you could provide an advanced graphics setting in the game, so that like minded enthusiasts could see what sort of graphics a stronger phone is capable of?

If your game features no locomotion then parallax corrected cubemaps make little sense for example, as the parallax can only really be perceived when moving the camera around.

It's not all black or white, there are definitely shades of grey when it comes to adding graphical features to your gear vr games.

You‘re absolutely right that it depends. My stance is the same; any dev looking to try different looks for their game should feel free to try using cubemaps of varying complexity on their target devices. What’s important is for devs to do the required testing before pushing through with such techniques for their game.

My point was that Oculus' advice sounds very different. According to them, Gear VR devs should avoid cubemap-based visual effects (especially specular+bump maps) and stick with unlit textures with diffuse lightmaps. (More recently, Carmack suggested doing realtime specular with 1 directional light, but no bump maps.)

Personally, I'm glad devs like you go against the grain; that's how we learn what works in games. 🙂 I also have tremendous respect for devs that push the limits of Gear VR's capabilities. Your work helps devs make bigger, better games that just wasn't feasible before.

[Norman3D]

Here is a comparison screenshot between the low quality materials and the high quality materials. Unfortunately a screenshot doesn't really make it justice, as things like parallax corrected cubemaps can mostly be appreciated when flying around the levels. You notice how the cubemap does not just "follow" your head, but is actually anchored to the world, giving that extra depth cue. Things like normal maps also look much more impressive in VR than on a still image.

Low quality: No normal maps, only cubemaps for metallic surfaces with roughness texture but no normal map and no parallax correction.


High quality: All features turned on 90% of the time, cause why not!



I take Oculus advice regarding specular as a much needed generalization, as I think most people have trouble getting there games to even reach 60fps on mobile on the first place. We on the other hand, knew getting to 60fps was the easy part, so I took it upon myself to put in as many graphical features as I could, seeing as most games on GearVR look very flat and don't seem to have much effort put into shader and shader optimization. And after all, I've been optimizing for mobile for quite a few years at this point, so it was a matter of using the tricks I developed along the way and it sure did pan out well!

EDIT: There is even more room for optimization, quite a bit of instructions are being used in the shader to decode the HDR lightmap that is encoded in a LDR texture. Unreal has support for ASTC texture compression, but they forgot or chose not to implement HDR compression (which is totally supported by ASTC) forcing me to use precious shader instructions instead. 😞

[firagabird]

Norman3D said:

[...] I took it upon myself to put in as many graphical features as I could, seeing as most games on GearVR look very flat and don't seem to have much effort put into shader and shader optimization.

Yes! Thank you!

This issue has bugged me mercilessly from the very first time I noticed it. Everything is shaded such that they look flat & dull. Some of the best looking games are able to minimize the effect of completely static lighting, but it has become almost mobile VR's visual fingerprint when the world lacks dynamic lights and/or reflections. It has reached the point where I wished I never realized it in the first place. This must be how John Carmack feels about aliasing (but I also feel the same way 😞 ).

There are other bits in your post I'd like to discuss further, but let me just give a big huge thank you for making dynamic lighting in mobile VR a defining highlight (no pun intended) of Daedalus. I'll have to borrow an S8 to see it in its full, parallax+normal mapped glory, but even just the metallic reflections and imperceptible aliasing on my S7 Exynos is enough for me to tell to everyone I know: Daedalus is the prime example of how good a mobile VR title can look in motion.