Intermittent GPU Hangs - No Application Crash
My team is working with Meta's branch of the UE Source Engine.
UE Version 5.2.0
Release 1.87.0
Oculus SDK version 55.0
Over the holidays, roughly coincidental with the v60 firmware release, we've been experiencing a very strange bug where audio will continue, but the frame hangs and the GPU crashes - sometimes for minutes at a time, and then will occasionally render a single frame, or a second or two of frames before hanging again repeatedly. but the application itself never crashes, and pending audio files will play if/when the GPU springs back to life.
We're testing and experiencing this bug on Quest 2/Pro/3's
We've tried a number of things, but I thought I'd put feelers out and see if anyone else has had similar issues or good suggestions on how to isolate a root cause.
16:24:33.759
OsSdk
failed to extrapolate controller tracking state.
16:24:34.173
Tracking
EXPCTRL: [Throttled count=44] Failed to find exposure settings in history.Setting the exposure commands is probably taking more than 3/4 frames
16:24:34.269
[CT]
Anchor:SlamAnchorMemoryOSSDKClient: [Throttled count=712] Skipping invalid anchorHandle when attempting to cache AnchorStates
16:24:35.190
Tracking
EXPCTRL: [Throttled count=45] Failed to find exposure settings in history.Setting the exposure commands is probably taking more than 3/4 frames
16:24:35.269
[CT]
Anchor:SlamAnchorMemoryOSSDKClient: [Throttled count=712] Skipping invalid anchorHandle when attempting to cache AnchorStates
16:24:35.536
adbd
failed to connect to socket 'tcp:6666': Connection refused
16:24:35.657
libosutils
Failed to open the file : error[-2]
16:24:35.657
libosutils
Failed to open the file : error[-2]
16:24:35.659
libosutils
Failed to open the file : error[-2]
16:24:35.660
libosutils
Failed to open the file : error[-2]
16:24:35.657
healthd
battery l=55 v=3865 t=23.5 h=2 st=2 c=155273 fc=5642000 cc=1 chg=a
16:24:35.666
libosutils
Failed to open the file : error[-2]
16:24:35.666
libosutils
Failed to open the file : error[-2]
16:24:35.773
OsSdk
failed to extrapolate controller tracking state.
16:24:36.190
Tracking
EXPCTRL: [Throttled count=44] Failed to find exposure settings in history.Setting the exposure commands is probably taking more than 3/4 frames
16:24:36.310
[CT]
Anchor:SlamAnchorMemoryOSSDKClient: [Throttled count=712] Skipping invalid anchorHandle when attempting to cache AnchorStates
16:24:37.190
Tracking
EXPCTRL: [Throttled count=44] Failed to find exposure settings in history.Setting the exposure commands is probably taking more than 3/4 frames
16:24:37.308
Tracking
CalibrationConversions: Serializing lookup table model by bypassing the normal procedure!
16:24:37.310
[CT]
Anchor:SlamAnchorMemoryOSSDKClient: [Throttled count=689] Skipping invalid anchorHandle when attempting to cache AnchorStates
16:24:37.540
adbd
failed to connect to socket 'tcp:6666': Connection refused
16:24:37.786
OsSdk
failed to extrapolate controller tracking state.
16:24:38.224
Tracking
EXPCTRL: [Throttled count=45] Failed to find exposure settings in history.Setting the exposure commands is probably taking more than 3/4 frames
16:24:38.338
[CT]
Anchor:SlamAnchorMemoryOSSDKClient: [Throttled count=712] Skipping invalid anchorHandle when attempting to cache AnchorStates
16:24:39.224
Tracking
EXPCTRL: [Throttled count=44] Failed to find exposure settings in history.Setting the exposure commands is probably taking more than 3/4 frames
16:24:39.242
Adreno-GSL
<log_gpu_snapshot:471>: panel.gpuSnapshotPath is not set.not generating user snapshot
16:24:39.246
OsSdk
Requested historical tracking state too far in the past; oldest: 12676.634840s, requested: 12289.658668s
16:24:39.249
kgsl-3d0
CP opcode error interrupt | opcode=0x00000000
16:24:39.295
libosutils
Failed to open the file : error[-2]
16:24:39.295
libosutils
Failed to open the file : error[-2]
16:24:39.296
healthd
battery l=55 v=3863 t=23.5 h=2 st=2 c=92773 fc=5642000 cc=1 chg=a
16:24:39.300
libosutils
Failed to open the file : error[-2]
16:24:39.300
libosutils
Failed to open the file : error[-2]
16:24:39.308
libosutils
Failed to open the file : error[-2]
16:24:39.308
libosutils
Failed to open the file : error[-2]
16:24:39.346
FenceChecker
FenceChecker: wait timed out after 100 msec
16:24:39.346
kgsl-3d0
MISC: GPU hang detected
16:24:39.346
kgsl-3d0
main[24710]: gpu fault ctx 6 ctx_type VK ts 9488 status 00800005 rb 0540/0b02 ib1 0000004007F7C000/027e ib2 00000040082826D0/02df
16:24:39.346
kgsl-3d0
main[24710]: gpu fault rb 2 rb sw r/w 0540/0b02
16:24:39.347
platform 3d6a000.qcom,gmu
Suspended GMU
16:24:39.351
kgsl-3d0
main[24710]: gpu skipcmd ctx 6 ts 9489 policy 80
16:24:39.353
kgsl-3d0
CP opcode error interrupt | opcode=0x00000000
16:24:39.360
kgsl-3d0
MISC: GPU hang detected
16:24:39.361
kgsl-3d0
main[24710]: gpu fault ctx 6 ctx_type VK ts 9500 status 00800005 rb 0600/0afc ib1 000000400873E000/027c ib2 00000040039BD000/079d
16:24:39.361
kgsl-3d0
main[24710]: gpu fault rb 2 rb sw r/w 0600/0afc
16:24:39.362
platform 3d6a000.qcom,gmu
Suspended GMU
16:24:39.366
kgsl-3d0
main[24710]: gpu skipcmd ctx 6 ts 9501 policy 80
16:24:39.370
Adreno-GSL
<log_gpu_snapshot:471>: panel.gpuSnapshotPath is not set.not generating user snapshot
16:24:39.377
kgsl-3d0
CP opcode error interrupt | opcode=0x00000000
16:24:39.384
kgsl-3d0
MISC: GPU hang detected
16:24:39.384
kgsl-3d0
main[24710]: gpu fault ctx 6 ctx_type VK ts 9513 status 00800005 rb 0b30/0c3b ib1 00000040089C5000/0280 ib2 0000004008BD1AB8/01fe
16:24:39.384
kgsl-3d0
main[24710]: gpu fault rb 2 rb sw r/w 0b30/0c3b
16:24:39.384
platform 3d6a000.qcom,gmu
Suspended GMU
16:24:39.393
[CT]
Anchor:SlamAnchorMemoryOSSDKClient: [Throttled count=712] Skipping invalid anchorHandle when attempting to cache AnchorStates
16:24:39.388
kgsl-3d0
main[24710]: gpu skipcmd ctx 6 ts 9514 policy 80
16:24:39.418
MediaCodec
mapFormat: no mediaType information
16:24:39.447
Adreno-GSL
<log_gpu_snapshot:471>: panel.gpuSnapshotPath is not set.not generating user snapshot
Looking over the logs and errors, I am seeing a lot of 'Failed at frameworks/native/libs/binder/PersistableBundle.cpp:168 (erase)'
and we've seen an error thrown in the OVRPlugin about "Unexpected Event Type 304" in Compositor::PollEvent from OVRPlugin/Src/Util/Compositor.cpp
We also have seen some logs mentioning GPU page faults...
We're in the process of testing just upgrading engine versions to see if it's some issue that's quarantined to v55.