Passthrough camera data is not available on Quest 3 developing with Unity

Participant

metalex201, thanks for sharing all of this information. It is unfortunate that even with all if that you can't get raw camera access.

Metalex and PhDog , have you guys tried playing around with Meta's Ocean Computer Vision Framework, even though it requires an external camera connected to the Quest? -https://facebookresearch.github.io/ocean/docs/demoapps/questapps/externalcamera/

It isn't perfect for what we need access to, using onboard cameras, but it may help.

Looking on the project's GitHub repo, the developers are actively working on the project, so hopefully Quest support will expand. There is some information on building for Quest's Android system, which may give you some insights on system access via ADB, since they also address requesting camera access permissions, even if for external cameras.

Building for Quest: https://github.com/facebookresearch/ocean/blob/main/building_for_meta_quest.md

Building for Android: https://github.com/facebookresearch/ocean/blob/main/building_for_android.md#1-prerequisites

Hopefully Meta allows developers raw camera access in the future, even if apps are not publishable to their store as a first step, but in the meantime the Ocean Framework may help.

metalex201

Participant

Thanks for the info, but sadly external camera access is a seperate from the passthrough access, you can request access for external usb cameras, that is possible but thats done through usb access which is different than accessing internal cameras, where the internal cameras are just missing when calling any camera code from an android app, possibly because the internal camera workings of the quest is very barebones so that only system processes can have access to it.

Participant

I know this isn't image processing, but it is an interesting use of the depth api on the Quest: https://www.linkedin.com/posts/marlene-j%C3%BCngling-a8a6322a3_quest3-depthapi-project-finally-activity-7225174863039090688-_bY4?utm_source=share&utm_medium=member_android

If we had access to the RGB camera we could do this and more, imagine mapping virtual objects to physical ones or providing better depth occlusion based on dynamic physical objects.

metalex201

Participant

I'm so close, yet so far 😧

this is from the package named "qrscanner" which is responsible for the UI and function of the qr scanner used at startup and for wifi code scanning. This is specifically from the library named "libqrcodereader_jni.so"

The app itself uses a class called XrSessionBase, which if I'm correct then it somehow runs an empty VR app in a "virtualDisplay" and uses it as a surface that is then called by this "libqrcodereader_jni.so"

sadly I couldnt get this to work on a decompiled version of the app, because "libqrcodereader_jni.so" is a system library, which is not accessible, So I'm gonna try something else, maybe create a virtualDisplay that just has an empty unity XR app whit passthrough turned on, and if i can somehow get that to run under an application then i could just get the surface from that virtualdisplay and I could use that as basically a raw passthrough layer for an open API, althought thats another roadblock since I have never used unity for AR, and I don't even know how I would run android code from there ._.

Participant

This is good metalex201 , getting closer.

I just saw this article about another way to get camera access which might help: https://www.uploadvr.com/quest-3-raw-camera-access-better-workaround/

The app makes use of Android's Media projection (https://developer.android.com/media/grow/media-projection?ref=uploadvr.com) and the demo shows the use of the AprilTag as an AR marker to place a virtual object in relation to it and then manipulate its pose (move the marker around).

The project's GitHub Repo: https://github.com/trev3d/QuestDisplayAccessDemo?ref=uploadvr.com

metalex201

Participant

I didn't know media projection worked on quest, thanks for mentioning that It will be usefull, althought I'm probably gonna try through ADB to get virtualdisplay working so I can avoid the issues that were listed on github, like media projection needing to be restarted.

NeedForLightSpeed

Up and Coming

Thank you Meta !!

We finally have a Passthrough API coming 😄

Announced at Meta Connect 2024

florianbuchholz1988

Inspiring

Long overdue, but yes. It's happening. No surprise really that it was announced now, since they had to fix that with the AVP as the competitor. Let's see how much access we are really getting.

marcussacana

Participant

It seems I can start updating my quest again, I was really holding the FW for any possible exploit that would be used to root the device and unlock basic features.

Participant

For those that want to relive the moment it was finally announced, the Camera "Passthrough API" was announced today, September 25th, 2024 and is coming in early 2025.

At around 1:56:30: https://www.facebook.com/MetaforDevelopers/videos/449444780818091
"This will enable all kinds of cutting edge MR experiences. You got tracked objects, AI applications, fancy overlays, scene understanding, and so much more".

Let's see what we will actually be given access, since not much was disclosed aside from this phrase about it. All of the things mentioned could be done using different AI models Meta has shown and not only via direct camera access, so hopefully they actually give is direct/raw RGB camera access and access to the depth camera/map as well.

I haven't checked in detail, but I wonder if we will learn more about this API in one of the other smaller talks/developer sessions.

seeingwithsound

Participant

To jump right to the camera passthrough API part of the video
https://www.facebook.com/watch/?v=449444780818091&t=7085

Seeing with Sound - The vOICe http://www.seeingwithsound.com