Hand tracking outdoors
Hi guys, We are currently experiencing an issue with hand tracking in our app when using the Quest Pro outdoors. The symptoms: The hand tracking seems to get into some state where it is unusable - the hand related pointers inside the app are flickering and unusable, and even the native palm pinch gesture for quick actions is flickering and unusable - see gif below, where I just hold my fingers in the close to pinched state that will normally just show a non-flickering icon (and will normally work if fully pinched). This unusable state of the hand tracking just continues, and the app/device does not really seem to be able to recover from it - usually a restart is needed. When does it happen? It only ever happens when being outdoors, and it seems to occur more often when there is sunshine. Maybe it occurs even more frequently when moving between shade and sunlight, but this has not been properly tested. It seems to be triggered by opening our application - we don't experience it when just staying in the OS menus. Also, it doesn't occur on every run of the application. Sometimes we can run it a couple of times (after restart) where the hand tracking is working fine (even in the same difficult light settings), but then, maybe on the third run or so, it comes into the flickering state. Sometimes it comes into the flickering state on the first run. What is our setup? We are developing our app in Unity (version 2022.3.2.f1) and we are using the currently latest version of the Meta XR Utilities package (com.meta.xr.sdk.utilities) version 55.0.0-preview. We are also using MRTK version 2.8.3. Our app uses passthrough. The headset is, as mentioned, Meta Quest Pro and it runs the currently lates OS (build 56.0). Some clues and thoughts: We assume that the issue is related to bright outdoor lighting and exposure settings of the cameras/sensors. When looking at the logs via ADB we see some potential clues as to what might be happening: On the runs where the hand tracking of the application does not get into the unusable, flickering state, log entries like these occur only a couple of times: 14:26:00.648 2800 3105 I ServiceInputManager: HANDTRACKING hand 0 status set to TooBright 14:26:00.648 2800 3105 I ServiceInputManager: HANDTRACKING hand 1 status set to TooBright 14:26:00.648 2800 3105 I ServiceInputManager: HANDTRACKING: Setting brightness flag to fire to 2 14:26:00.648 2800 3105 I ServiceInputManager: HANDTRACKING SendHandsBrightnessWarning 14:26:00.648 2800 3105 I ServiceInputManager: HANDTRACKING trying to find the function to call 14:26:00.648 2800 3105 I ServiceInputManager: HANDTRACKING Really sending SendHandsBrightnessWarning And after each time, it seems that we get (as a kind response) logs like this: 14:26:00.719 1014 1150 I TrackingService: [ ExposureControl] HandTrackingUseIOT -> HandTracking transition time: 73 ms though sometimes it seems that the transition is the other way around (from HandTracking to HandTrackingUseIOT). And another 'response' seems to be logs like these: 14:26:00.633 1014 3854 I TrackingService: setMuxModeFeatureFidelity: Changing active camera mux mode to HandTracking 14:26:00.633 983 1265 D CameraMuxModeService: Switching active CameraMuxMode from 'HandTrackingUseIOT' to 'HandTracking' On the runs where the hand tracking of the application does get into the unusable, flickering state, we get the first block of logs that sends the SendHandsBrightnessWarning a lot of times (more or less constantly). And we don't seem to get the 'responses' to these logs, like we do when the hand tracking is working properly. So it seems like there are some services that react to the SendHandsBrightnessWarning when the hand tracking is working properly and these services do not react when we experience the flickering unusable state... We have tried to set the Hand Tracking Support to Hands Only (rather than Controller And Hands) on the OVRManager.cs but it does not seem to make a difference. Does anyone have an idea of how to fix this issue? If so, we would be very grateful. Kind regards, Jakob
