Re: Hand tracking outdoors - Meta Community Forums

jakobanarkylabs · ‎08-21-2023

Hi guys,

We are currently experiencing an issue with hand tracking in our app when using the Quest Pro outdoors.

The symptoms:
The hand tracking seems to get into some state where it is unusable - the hand related pointers inside the app are flickering and unusable, and even the native palm pinch gesture for quick actions is flickering and unusable - see gif below, where I just hold my fingers in the close to pinched state that will normally just show a non-flickering icon (and will normally work if fully pinched). This unusable state of the hand tracking just continues, and the app/device does not really seem to be able to recover from it - usually a restart is needed.
Even the native palm pinch gesture goes into an unusable, flickering state when the issue occurs

When does it happen?
It only ever happens when being outdoors, and it seems to occur more often when there is sunshine. Maybe it occurs even more frequently when moving between shade and sunlight, but this has not been properly tested.
It seems to be triggered by opening our application - we don't experience it when just staying in the OS menus. Also, it doesn't occur on every run of the application. Sometimes we can run it a couple of times (after restart) where the hand tracking is working fine (even in the same difficult light settings), but then, maybe on the third run or so, it comes into the flickering state. Sometimes it comes into the flickering state on the first run.

What is our setup?
We are developing our app in Unity (version 2022.3.2.f1) and we are using the currently latest version of the Meta XR Utilities package (com.meta.xr.sdk.utilities) version 55.0.0-preview. We are also using MRTK version 2.8.3. Our app uses passthrough. The headset is, as mentioned, Meta Quest Pro and it runs the currently lates OS (build 56.0).

Some clues and thoughts:
We assume that the issue is related to bright outdoor lighting and exposure settings of the cameras/sensors. When looking at the logs via ADB we see some potential clues as to what might be happening:

On the runs where the hand tracking of the application does not get into the unusable, flickering state, log entries like these occur only a couple of times:

14:26:00.648  2800  3105 I ServiceInputManager: HANDTRACKING hand 0 status set to TooBright
14:26:00.648  2800  3105 I ServiceInputManager: HANDTRACKING hand 1 status set to TooBright
14:26:00.648  2800  3105 I ServiceInputManager: HANDTRACKING: Setting brightness flag to fire to 2
14:26:00.648  2800  3105 I ServiceInputManager: HANDTRACKING SendHandsBrightnessWarning
14:26:00.648  2800  3105 I ServiceInputManager: HANDTRACKING trying to find the function to call
14:26:00.648  2800  3105 I ServiceInputManager: HANDTRACKING Really sending SendHandsBrightnessWarning

And after each time, it seems that we get (as a kind response) logs like this:

14:26:00.719  1014  1150 I TrackingService: [   ExposureControl] HandTrackingUseIOT -> HandTracking transition time: 73 ms

though sometimes it seems that the transition is the other way around (from HandTracking to HandTrackingUseIOT).
And another 'response' seems to be logs like these:

14:26:00.633  1014  3854 I TrackingService: setMuxModeFeatureFidelity: Changing active camera mux mode to HandTracking
14:26:00.633   983  1265 D CameraMuxModeService: Switching active CameraMuxMode from 'HandTrackingUseIOT' to 'HandTracking'

On the runs where the hand tracking of the application does get into the unusable, flickering state, we get the first block of logs that sends the SendHandsBrightnessWarning a lot of times (more or less constantly). And we don't seem to get the 'responses' to these logs, like we do when the hand tracking is working properly.

So it seems like there are some services that react to the SendHandsBrightnessWarning when the hand tracking is working properly and these services do not react when we experience the flickering unusable state...

We have tried to set the Hand Tracking Support to Hands Only (rather than Controller And Hands) on the OVRManager.cs but it does not seem to make a difference.

Does anyone have an idea of how to fix this issue? If so, we would be very grateful.

Kind regards, Jakob

AliceinVRworld · ‎08-23-2023

Hey there....So the reason you're having this issue is because the VR devices should never be used outdoors. It is meant for inside your home or overall building. Being outdoors especially with sun could cause sun damage on the screen.

jakobanarkylabs · ‎08-24-2023

Yes, I am aware that the Quest Pro is not really meant to be used outdoors - it tells me so in the 20 sec obligatory safety animation. And if the hand tracking was failing every time, I would not pursue this any further.

However, it seems quite often to be working fine - also in the exact same sunny conditions. If I am not opening our app (just being in the operating system environment) the issue does not seem to occur - and on some runs of our app, it also works just fine.
The logs that I posted also seem to suggest that there are some services that do not react whenever we are experiencing the issue.

So can you rule out the possibility that there is something that could be done to fix this issue - either in our app code/configuration, the OpenXR backend or at the OS level - or is it more a question of Meta having decided not to support any outdoor usage / usage in brightly lit areas?

Regarding the possible damage to the screen, how exactly does the damage occur? Or is it more a camera/sensor issue? So far, I haven't experienced anything that suggests lasting physical damage, but of course that might happen - I am willing to take that risk.

AliceinVRworld · ‎08-27-2023

I would not be able to provide you with the reason why it works with certain days. These devices are not meant for outside because of the sun damage it could cause. I personally think since there are moving stuff outside the cameras will loose their tracking when playing games and you will lag on games.

The camera's are still working with early stages on vr as there is more that could come in the future. However, it needs to track your environment to remember your play area and work as a vr device as you're still and in a closed space. To properly capture the vr experience (is just how it works).

Like I mentioned before please don't use this device outside you could make it have sun damage on lens and screen. Having bright light on the device can cause sun damage by making burns on the screen when using it the device with sun directing it.

Like the image below.

zphill15 · ‎12-13-2023

Hi Alice,

Is there any way to sun proof the device so that we can use the device outdoors. That would be ideal for my use case. Like a protective skin or film that we could place over the device that works similar to sunglasses to block it from the damaging parts of the light spectrum.

AliceinVRworld · ‎12-13-2023

I am not sure about this normally these devices are not for sun/outdoors. Unless there is something not approved by Meta but currently there is not.

jakobanarkylabs · ‎05-15-2024

Ok, just a little update on this topic.

Since I made the original post, we have switched a bit away from using the Quest Pro and we are now mainly using the Quest 3 instead.
We were also initially experiencing some flickering of the hand tracking on the Quest 3 when using our app outdoors in bright conditions. But after some testing, we figured out that it seemed to be caused by our setting of hand tracking frequency which was set to high.

First we couldn't figure out why the resulting AndroidManifest.xml in the built apk would still have com.oculus.handtracking.frequency set to HIGH even when changing the settings on the OVRManager to LOW. But that was due to the fact that the custom AndroidManifest.xml we had in Assets/Plugins/Android still had that setting (we had previously created it by choosing the menu Oculus->Tools->Create store-compatible AndroidManifest.xml). But either deleting the custom AndroidManifest.xml or re-creating it after the changed setting of hand tracking frequency solved that problem.

With the hand tracking frequency set to LOW in the apk's AndroidManifest.xml, or not set at all, we don't experience the hand tracking flicker. And we don't really notice any difference in the hand tracking responsiveness - at least in our use of it.

<meta-data android:name="com.oculus.handtracking.frequency" android:value="LOW" />

An extra interesting finding: there is a developer setting on the Quest (in System/Developer) called Hand Tracking Frequency Override, and if we set that to High, we see the hand tracking flickering even in the operating system menus (outdoors in bright conditions).