Android Oreo breaking shaders/textures

[Norman3D]

Hello everybody,
we are getting more and more reports of users having issues with Daedalus. Specifically, certain elements in the game are being rendered pitch black.
Here are two screenshots:



I've personally been able to confirm this happening on a Note 8 with Exynos. Game works fine on Android Nougat, updating to Android Oreo causes the issue. Other users have claimed the same. I'm not sure if this is an Exynos-only problem or not. I only have access to Exynos devices here in Spain.
The amount of complaints rise steadily as the Oreo update is being pushed to users.

There are also liquid platforms in the game, featured for the first time in the third level of stage 3 that are also experiencing this problem.
I'm not entirely sure if it's a problem with the shader, as you can see, the white elements in the wand still work (first picture). I believe it must be the cubemap that is being passed to the shader as black, which is then multiplying with the diffuse color, resulting in a black surface. The white part you see on the wand is added at the end of the shader. That would explain why it's still visible.

It might also be interesting to note, that Daedalus uses OpenGL ES 3.1 to make use of ASTC compression, prefiltered cubemaps, and uncompressed 8-bit RGB textures sampled in the vertex shader to make the liquid platforms possible.

The current build uploaded to the Store is running Unreal 4.16 I believe, but we have another build in a private beta channel with 4.18. The issue is present in both.

I'm also attaching to this thread two logs. One taken with Android 7, and the other one with Android 8.
The main differences I can find between the logs are:

Slightly different GL_VERSION:
Android7: GL_VERSION: OpenGL ES 3.2 v1.r3p0-00rel0.dce8ca6a85819d8495118734e2e3ccc6
Android8: GL_VERSION: OpenGL ES 3.2 v1.r9p0-01rel0.1af9af8f7e4030fa12a53451ae1c788c

Different GL_MAX_TEXTURE_SIZE: (I don't think this is relevant to us, no texture exceeds 2048 in our game)
Android7: GL_MAX_TEXTURE_SIZE: 16384
Android8: GL_MAX_TEXTURE_SIZE: 8192

It is also interesting to note that there is an error message present in both logs:
Error: Unsupported EPixelFormat 28
That error is apparently related to 32bit per channel textures, which we make no use of. :s

Any help is greatly appreciated as we feel a bit lost with this one.

Thanks!

[Norman3D]

It just occurred to me to try the Daydream version of the game and similar thing is happening. Logo and wand turn out black, but the liquid platform works! However, the ondulated platform that is spawned right after (which is a totally different mesh/shader) does not work, whereas un gearvr it does!
I'm not entirely sure if these disparities between GearVR and Daydream mean it's a driver issue or something else on the platform side.

[Norman3D]

imperativity said:

@Norman3D

Thanks for the additional information here. I am passing this along internally.

Are you using multiview perchance?

Thanks!
The 4.18 version is using multiview but the 4.16 version isn't. The error is present in both cases, though.

[Norman3D]

imperativity said:

@Norman3D

An update for your from our integrations team:

"this is an issue at the driver level and we're not gonna be able to change that driver, so the only thing we can reliably do here is find a workaround and update the app. 

They need to play around with UE and figure out what works and what doesn't in terms of texture formats and sampling types...

Can they try something else other than ASTC for that texture, like uncompressed or ETC2?"

Hi @imperativity ! Thanks a lot for the info!
I also came to the conclusion that it had to be driver bug. I have since been able to find some time to debug this issue by trial and error. I'm still looking into it, but it seems that the driver bug has some sort of "rounding error". Not sure if it's a feature or a bug.

At the beginning of Daedalus production, the shader had a parameter value called "Reflection Intensity" that I would adjust manually for each material instance for it to look "right". These values, due to how the shader was built at the time, required high values such as 100 to 300. Later in development the base shader changed, and it required much lower "Reflection Intensity" values. Due to the fact we had only 2 months to develop the game from start to finish, I had no time to go through all materials (they are a lot) to lower the values manually and instead opted for adding one more instruction to the shader that would multiply this "Reflection Intensity" value by "0.05".
So far so good. If you output this result in Android Oreo, everything is fine. If you do anything else in the shader after the 0.05 multiplication, it will round "0.05" to "0.0", making the result look black. The fix consists of pre-multiplying the "Reflection Intensity" value and removing the "0.05" multiplication.

Keep in mind that all shaders in the game have the same parent material, so they all have this "0.05" multiplication, however not all of them showcased the rounding bug. Applying this fix on all of them, fixes those shaders with the problem, but the rounding error shows up on others. I have not been able to figure out exactly why it happens. For now I'm working around the black shader issue by applying the fix only on those shaders showcasing the issue and ignoring the rest. Makes me feel uneasy, but it works.

I'll post another update if I find out more information!

[firagabird]

Just wanted to chime in at how frustrating it is when a game breaks because of a driver bug introduced by a new version of Android. It's infuriating both because you know everyone's gonna upgrade to it because it's the Shiny New Thing^tm, and yet driver patches will take forever if at all to fix the regression.

The Oreo update is looming over my S7; I'll be holding out until my phone dies. Or when I can get my hands on Go. 😉

[Norman3D]

imperativity said:

@Norman3D

Here is some feedback from our integrations team:

"When dealing with floats in mobile shaders you can select a precision level, lowp/mediump/highp. Could they try putting that ReflectionIntensity to highp to see if that helps them? 

Again, them providing us with a minimal repro (just sending the .uasset of the material that has the bug) helps us in getting a driver fix."

That is something Carmack mentioned in his public critique of Daedalus as well. The frustrating part is that I have no control over precision qualifiers for variables in Unreal. I don't understand why as it would be as simple as letting me choose the precision qualifier for each variable node, but such option does simply not exist. 😞

Unfortunately, all precision qualifiers in Unreal are set to highp by default. At least if you are using OpenGL ES 3.1 it seems, that was at least Carmack's observation of the compiled shaders in Daedalus. However, things have changed in some version of Unreal that was released since Daedalus first launched, and unless the "Use full precision" checkbox is marked in the material setting, a lower precision is used for all variables (or at least that what it claims to do). So that's the good news, the bad news is that this does not fix the above mentioned issue, perhaps because it's actually not doing that. (of course it's still an issue that I can't be the one to decide what gets highp and what gets lowp and instead have to choose whether everything gets highp or if everything gets lowp, but I digress)

In regards to providing you guys with a minimal repro, let me work on that. I'll PM you a dropbox link soon.