Deadlocks on the GPU

[KamilWenkly]

Hi! 
Recently, we have been encountering another significant problem that we are struggling to resolve.
Elven Assassin on Oculus Quest 1/2 stops rendering new frames during gameplay and after some time it throws players to Oculus Home.

From our research, it seems that this is caused by deadlocks on the GPU.
Here is the error code from Adreno-GSL that we are getting: 0x400c0907
Interestingly, the problem does not occur on all devices, but when it occurs, it is always on the same ones.

Below you will find a fragment of the crash log for you to see.
The most important part starts at line 10, where you can see the mentioned error code as well as information that deadlock would occur.
We would like to know what can cause such deadlocks as we cannot reproduce them and how to address them.

09-11 20:11:33.763   841  5415 W Memory Allocation: large allocation 4001056
09-11 20:11:33.786   841  5415 W Memory Allocation: large allocation 4524304
09-11 20:11:34.191  1456  2866 I [CT]    : MIXEDREALITY: MrServiceView: renderIteration: (every 10 seconds), screen is: ON, PT is: OFF, ID is: OFF
09-11 20:11:34.266  1916  2954 I Telemetry: App memory usage: PSS=62MB DalvikPSS=20 MB
09-11 20:11:34.454  1828  2874 D [OAO] VrNotificationFeedManager: Notification feed fetch success callback, scheduling next feed call now.
09-11 20:11:34.651  3556  5500 I VrApi   : FPS=9/72,Prd=42ms,Tear=9,Early=0,Stale=9,VSnc=1,Lat=1,Fov=0,CPU4/GPU=2/4,1651/670MHz,OC=FF,TA=0/0/0,SP=N/N/N,Mem=1554MHz,Free=537MB,PLS=0,Temp=35.0C/0.0C,TW=0.00ms,App=0.00ms,GD=0.00ms,CPU&GPU=59.08ms,LCnt=1,GPU%=1.00,CPU%=0.00(W0.00),DSF=1.00
09-11 20:11:34.750  1916  2621 W OsSdk   : Requested historical tracking state too far in the past; oldest: 6397.378507s, requested: 6312.063881s
09-11 20:11:34.761   841  5498 I [CT]    : HealthReport: Everything Passed =)
09-11 20:11:34.782  1916  2621 W OsSdk   : Requested historical tracking state too far in the past; oldest: 6397.414430s, requested: 6312.063881s
09-11 20:11:34.860  3556  3605 W Adreno-GSL: <gsl_ldd_control:552>: ioctl fd 47 code 0x400c0907 (IOCTL_KGSL_DEVICE_WAITTIMESTAMP_CTXTID) failed: errno 35 Resource deadlock would occur
09-11 20:11:34.860  2134  5436 I TimeWarp: Tear: 6.823913 (since wakeup: 6.823892)
09-11 20:11:34.869  3556  3605 W Adreno-GSL: <log_gpu_snapshot:461>: panel.gpuSnapshotPath is not set.not generating user snapshot
09-11 20:11:34.869  3556  3605 W Adreno-GSL: <gsl_ldd_control:552>: ioctl fd 47 code 0xc0200933 (IOCTL_KGSL_TIMESTAMP_EVENT) failed: errno 22 Invalid argument
09-11 20:11:34.872  3556  3605 W Adreno-GSL: <ioctl_kgsl_syncobj_create:4876>: (30, 24, 398142) fail 22 Invalid argument
09-11 20:11:34.898  3556  3605 W VrApi   : FrameIndex = 379138, RetiredClientFrameIndex = 379133, waited for 25 msec for frame to retire
09-11 20:11:34.901  2134  5436 W OsSdk   : Requested historical tracking state too far in the past; oldest: 6397.531188s, requested: 6390.976662s
09-11 20:11:34.902  2134  5436 I TimeWarp: Tear: 6.858338 (since wakeup: 0.000783)
09-11 20:11:34.902  2134  5436 I Vsync   : Vsync 6595: 494 x 13.92 ms (71.82 Hz, at 6397.821)
09-11 20:11:34.902  2134  5436 I TimeWarp: Adjusting 6595.58 > 6103 to 6596
09-11 20:11:34.902  2134  5436 I TimeWarp: SetSchedFifo( tid=5505, pol=3, pri=0 ) succeeded
09-11 20:11:34.915  2134  5436 W TimeWarpClient: FetchFrame: no more room for another frame for client(8)
09-11 20:11:34.916  3556  3605 W Adreno-GSL: <gsl_ldd_control:552>: ioctl fd 47 code 0xc0200933 (IOCTL_KGSL_TIMESTAMP_EVENT) failed: errno 22 Invalid argument
09-11 20:11:34.916  3556  3605 W Adreno-GSL: <ioctl_kgsl_syncobj_create:4876>: (30, 24, 398142) fail 22 Invalid argument
09-11 20:11:34.921  3556  3605 E Unity   : OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_OPERATION: Operation illegal in current state
09-11 20:11:34.921  3556  3605 W Adreno-GSL: <gsl_ldd_control:552>: ioctl fd 47 code 0xc040094a (IOCTL_KGSL_GPU_COMMAND) failed: errno 35 Resource deadlock would occur
09-11 20:11:34.921  3556  3605 W Adreno-GSL: <log_gpu_snapshot:461>: panel.gpuSnapshotPath is not set.not generating user snapshot
09-11 20:11:34.921  3556  3605 W Adreno-GSL: <gsl_ldd_control:552>: ioctl fd 47 code 0xc0200933 (IOCTL_KGSL_TIMESTAMP_EVENT) failed: errno 22 Invalid argument
09-11 20:11:34.921  3556  3605 W Adreno-GSL: <ioctl_kgsl_syncobj_create:4876>: (30, 24, 398142) fail 22 Invalid argument
09-11 20:11:34.922  2134  5436 I Vsync   : Vsync 6597:  2 x 13.92 ms (71.82 Hz, at 6397.842)
09-11 20:11:34.926  3556  3579 E OsSdk   : failed to extrapolate state.
09-11 20:11:34.928  3556  3579 I chatty  : uid=10004(com.WenklyStudio.ElvenAssassin) UnityMain identical 8 lines
09-11 20:11:34.928  3556  3579 E OsSdk   : failed to extrapolate state.
09-11 20:11:34.948  3556  3605 E CRASH   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-11 20:11:34.949  3556  3605 E CRASH   : Version '2019.4.20f1 (6dd1c08eedfa)', Build type 'Release', Scripting Backend 'il2cpp', CPU 'arm64-v8a'
09-11 20:11:34.949  3556  3605 E CRASH   : Build fingerprint: 'oculus/vr_monterey/monterey:10/QQ3A.200805.001/18371800230900000:user/release-keys'
09-11 20:11:34.949  3556  3605 E CRASH   : Revision: '0'
09-11 20:11:34.949  3556  3605 E CRASH   : ABI: 'arm64'
09-11 20:11:34.949  3556  3605 E CRASH   : Timestamp: 2021-09-11 20:11:34+0200
09-11 20:11:34.949  3556  3605 E CRASH   : pid: 3556, tid: 3605, name: Thread-3  >>> com.WenklyStudio.ElvenAssassin <<<
09-11 20:11:34.949  3556  3605 E CRASH   : uid: 10004
09-11 20:11:34.949  3556  3605 E CRASH   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
09-11 20:11:34.949  3556  3605 E CRASH   : Cause: null pointer dereference
09-11 20:11:34.949  3556  3605 E CRASH   :     x0  0000000000000000  x1  0000000000000000  x2  0000000000001080  x3  0000000000000000
09-11 20:11:34.949  3556  3605 E CRASH   :     x4  0000000000001080  x5  000000759bd41588  x6  0000000034155555  x7  000000000012d5dc
09-11 20:11:34.949  3556  3605 E CRASH   :     x8  000000763752b800  x9  000000759beeb220  x10 00000000000011a8  x11 0000000000000000
09-11 20:11:34.949  3556  3605 E CRASH   :     x12 0000000000000001  x13 0000000000000001  x14 00000074808f08b0  x15 0000000000000008
09-11 20:11:34.949  3556  3605 E CRASH   :     x16 000000754f6231e0  x17 0000007635840180  x18 0000000000008006  x19 00000074c05ed100
09-11 20:11:34.949  3556  3605 E CRASH   :     x20 0000000000003419  x21 0000000000000002  x22 0000000000001080  x23 0000000000000000
09-11 20:11:34.949  3556  3605 E CRASH   :     x24 0000000000008f37  x25 000000754f2fb75c  x26 000000754f6a7000  x27 00000074c05ed100
09-11 20:11:34.949  3556  3605 E CRASH   :     x28 0000007460000388  x29 0000007471f65c80
09-11 20:11:34.949  3556  3605 E CRASH   :     sp  0000007471f658b0  lr  000000754e8e2b50  pc  000000763584020c
09-11 20:11:34.949  3556  3605 E CRASH   : 
09-11 20:11:34.949  3556  3605 E CRASH   : backtrace:
09-11 20:11:34.950  3556  3605 E CRASH   :       #00 pc 000000000007d20c  /apex/com.android.runtime/lib64/bionic/libc.so (je_malloc_tsd_boot1+392) (BuildId: a6e0ca3a9989715134d1d1e2126e0f7a)
09-11 20:11:34.950  3556  3605 E CRASH   :       #01 pc 00000000008d2590  /data/app/com.WenklyStudio.ElvenAssassin-ms4aefglM4c2bxtuUEYNZQ==/lib/arm64/libunity.so (BuildId: 03350f4f3bf26860b774cd03fda0dab22b43049d)
09-11 20:11:34.950  3556  3605 E CRASH   :       #02 pc 00000000000d487c  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_attr_setdetachstate+24) (BuildId: a6e0ca3a9989715134d1d1e2126e0f7a)
09-11 20:11:35.027   841  5413 I [CT]    : VIPER: InternalAnchorHealthMonitor: Anchors: 1 | Persistent anchors: 1 | Anchors never observed: 0
09-11 20:11:35.027   841  5413 I [CT]    :   (min/avg/max) Anchor landmarks: (95|95.0|95) | Covisible keyframes: (39|39.0|39) | Times since observation: (0.0|0.0|0.0)
09-11 20:11:35.034  2134  5436 I TimeWarp: SetSchedFifo( tid=5505, pol=2, pri=0 ) succeeded
09-11 20:11:35.046   841  5410 I [CT]    : VIPER:LOCALIZER: Extracting non-gravity-aligned descriptors on train.
09-11 20:11:35.245   841  5415 W Memory Allocation: large allocation 4001056
09-11 20:11:35.274   841  5415 W Memory Allocation: large allocation 4131088

[martin.kucer]

I have the same issue. Any solution for this problem ?