Support for OpenXR standards eye and facetracking both native and through Oculus / Air Link

[Cain_Bloodbane]

We recently saw eye and facetracking getting support through Oculus Link and Air Link... but it was only by using proprietary Quest Pro specific OpenXR extensions. That means each and every developer then needs to add support for this, instead of Meta supporting the already existing OpenXR standards.

That means while we already have 10+ games that support eye tracking foveated rendering, the Quest Pro wont support that. Microsoft Flight Simulator... no eye tracking support there either. VRChat... again, no eye tracking or facetracking support. Unless each of these implement their own support for the Oculus specific extensions for something that is already implemented as a standard in OpenXR.

Why claim to support OpenXR to pull such a move? It makes no sense unless its some annoying attempt at taking control of the market by fragmenting it and hoping to win more users by it in the long run. I very much hope the current implementatioin is just a stepping stone towards implementing the OpenXR standards, sticking to the promises they already made to support OpenXR.

[coreynach]

Agree 100%

[Expir]

Have a good day at work

[BattleAxeVR]

Eye-tracking (on Pro) and body-tracking (on Quest 1 / 2 / Pro) works on PC over Link / AirLink now. I haven't tried face-tracking yet, but I presume it works too (on Pro)

[Cain_Bloodbane]

How did you test it? Did you test with games that use eye tracking or Quest Pro specific apps? Because when I made this there was already support for making apps that use eye and facetracking as long as you used the Oculus SDK and Quest Pro specific OpenXR extensions for eye and face tracking.

[BattleAxeVR]

For Body tracking on PC, they just posed a fix in the public beta branch that will come out in v47 apparently, and that works on all Quest 1, 2, and Pro. Which is a huge deal. I used my own test app to implement Waist-Oriented locomotion with the waist pose which works really well, btw, I encourage other devs to try it and steal the idea for their games.

https://forums.oculusvr.com/t5/OpenXR-Development/Waist-Oriented-locomotion-demo/td-p/1002044

[Cain_Bloodbane]

That does look nice, but it still seems to be Meta specific support. So it wont help Pro users f.ex. use Microsoft Flight Simulator in VR with eye tracking and dynamic foveated rendering because to use eye, face, body and hand tracking it needs Meta Quest specific OpenXR extensions.

If they made the Quest Pro support the OpenXR standards for eye, face and handtracking, there would be several games and apps that would support the Quest Pro.

It even shows it in the code. OpenXR_FB eye, face and body tracking. Pretty sure the hand tracking is also not the OpenXR standard, but I have not tried testing that.

[BattleAxeVR]

There's no reason Microsoft can't add code support for Meta-specific OpenXR extensions to MSFS, they already have Meta-specific controller bindings as does every other OpenXR game/app out there, and it's quite easy / trivial to add it.

I agree using generic hand / eye tracking extensions would make any app more broadly compatible with other eye/hand tracking hardware, but a company the size of Microsoft has no excuse.

It took me, a solo engineer, only a day to add eye-tracking to my engine where I can select items and shoot objects out of my eyes, or use it for dynamic foveated rendering. Anyway, both major engines, Unreal and Unity, support most Meta-specific extensions at this point (especially Unity. Unreal I haven't checked yet but they do use an Oculus-specific OpenXR plugin), such as changing refresh rate at runtime to override system-wide values per-title, or to set alpha blending flags for multi-layer apps.

For Hand tracking, it doesn't make sense to use the generic interface right now anyway, on Quest, which is why I didn't bother implementing it (yet), since I get all the bones in the hands as part of the body tracking, and this works whether using controllers or hand tracking currently, it switches automatically between them without a hitch. The body tracking extension gives you invaluable waist + elbow joints which, unless you use addons, requires the dev to implement it themselves. And in almost every VR game I've seen so far that tried to do full skeletal control and embodiment at a software level using only HMD + 2 controller poses, the elbows and knees would often suddenly flip backwards and contort into ridiculous, impossible poses, turning the entire game into a sad joke. Completely ruins immersion. So IMO taking advantage of the fantastic work that Meta provided us with here, solid elbow and waist + other bones, is worth the few hours of dev time it took me to hand-code it from scratch in C++. The docs are mostly fine for this, although there are no docs for link sharpening and local dimming yet I managed to get those working too (on standalone builds, I mean)

[Cain_Bloodbane]

Of course Microsoft could add it, I doubt they will because I do not think there are that many Quest Pro users, but what I find more important is that they should not have to. There are already standards in OpenXR for eye, face and handtracking. No reason for Meta to fracture the market even more than it is by not supporting existing standards. A company the size of Microsoft needs a reason to add the support, they obviously wont just support any VR headset that comes along that might support eye-tracking. They already do support eye-tracking.

Not sure why you mention the size of Microsoft when obviously Meta is also a big enough company to not have any excuse to not add support for the OpenXR standards.

Its nice that they have added body tracking, but it does not excuse Meta not using the OpenXR standards.

[BattleAxeVR]

I just wanted to say, I was totally wrong about the generic OpenXR extension support for eye tracking,

I need this as well as the FB/meta specific "social" eye-tracking extension provides smoothed out values which adds too much latency to be useful for foveated rendering (on PC). I was told this here by @johnkearney 

So eye-tracking is another semi-worthless feature for PC VR users and devs. The eye tracking social one is very neat though, and you can do a lot with it, as is the body tracking one (still lacks legs but that should come later I believe).

I don't need or expect Meta to actually fix this immediately, but just a confirmation that it WILL get done at some point is what will prevent me from returning my Quest Pro for a refund.