Forum Discussion
Passthrough camera data is not available on Quest 3 developing with Unity
Basically it is not possible to access the real time passthrough video from Quest 3 in Unity.
Many devs among the internet think it is a crucial feature to be able to process to create incredible mixed reality apps with object recognition using machine learning. Nevertheless, it is not possible to access the real time video captured by quest 3 in unity.
Here is a reddit post made by me where people are requesting this feature.
Please, we need access to real time passthrough video in Unity or other engines.
109 Replies
Meta added headset camera permission description to its documentation 3 days ago - https://www.meta.com/help/quest/511837681585017/
It mentions that users must allow camera access permission, which will allow the app to access the camera and what is in front of the headset (not including any virtual objects/content). The document provides some examples of possible camera uses (object tracking, location-based, and using machine learning to process camera feed). It also mentions access to real-time camera feed (which I am assuming and hoping means raw access) and intrinsic camera values (camera focal length, principal point, image dimensions and camera position relative to center of device).
I did not find any mention of camera access in their document on viewing and managing permissions (which was updated 5 days ago), so maybe there are still more things to be updated first before release: https://www.meta.com/help/quest/337925588963509/
At least it looks like the wait won't be too much longer. I'm hoping it comes out before the end of April or else it won't be early 2025 anymore.
Update: Meta will be giving some updates on their camera access at GDC.We finally now have camera access on the Quest 3(S) \o/
See documentation: https://developers.meta.com/horizon/documentation/unity/unity-pca-documentation
Sample project/code from Meta: https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples
A video has already been created showing how to access it in Unity: https://youtu.be/1z3pcMJbnRA?feature=shared
Tutorial on getting it working on a new/empty project: https://skarredghost-com.cdn.ampproject.org/c/s/skarredghost.com/2025/03/17/how-to-camera-access-meta-quest-3-unity-6/amp/
List of projects using camera access created by the community: Here is a list of what some developers have already created with the Quest camera access - https://github.com/xrdevrob/QuestCameraKit?tab=readme-ov-file#community-contributions
Current Limitations- No Unity Editor Camera Access: Unfortunately there isn't a way to access the camera via the Unity Editor, which was a useful feature with other AR devices, like the HoloLens, to speed up development.
- Apps can't be published on the official store: Applications can't be posted on Meta's store for now. Not sure if you can create it for desktop/pcvr though, in case you need to more processing and rendering power.
- Camera Resolution not Full-HD: The camera resolution isn't great, only 1280 x 960.
- Low/Mismatched Data Rate: Only 30Hz, much less than the refresh rate of what is being rendered and calculated.
- No Raw Depth Sensor Access? I'm not sure if we have raw access to the depth sensor, for instance for point cloud generation.
Hopefully these things will be improved in future software and hardware updates.
just commenting here so nobody forget about Unreal devs please!!!
passthrough API for unreal!!
Unfortunately there haven't been any updates on Meta's release of the camera API other than Andrew Bosworth (head of Meta's Reality Labs) confirming on Instagram (February 3rd, 2025) that it is still planned for early 2025.
In other news though, Google has confirmed that their upcoming OS Android XR will allow developers with camera access, which will work similarly to what is currently done on Android where the app requests camera access and the user must allow access. Though there is no confirmed release date for Samsung's Project Moohan headset, which will run on Android XR, it is expected to release this year.
Hopefully Meta releases their camera API soon or else they will be catching up to even more companies on this front, like Pico 4 Enterprise, Apple Vision Pro, the previous professional AR headsets (HoloLens 2, Magic Leap 2, Vuzix, etc.), all of which already allow camera access to mostly enterprise developers, and now Google/Samsung.
Hi all!
Are there any updates on the release of the Passthrough API or have I missed something?
CheersFinally...! Exciting moment!
- In the "Meet Meta Quest 3S" session, at around 3:37:30, the RGB cameras and sensors of the Quest 3 and 3S are discussed.
Both headsets have the same RGB cameras (RGB 18 PPD, 4MP) but are in a different arrangement but are said to be similar in passthrough quality. I assume this will also result in similar raw camera access quality between headsets if they do some blending beforehand, but if not this hopefully won't impact the image feed, and consequently the image processing quality too much.Quest 3 has a hardware depth projector (mainly used during room setup to improve detection and stability on plain/texture-less walls) while the 3S does not (it has a flicker sensor light pipe design and flood infrared illuminators, to improve hand tracking in low-light environments). It was mentioned that depth is calculated via the RGB cameras rather than the depth sensor, which is strange. Maybe the decision to use RGB for depth calculation instead of the depth sensor has to do with the depth sensor having a different and lower resolution and field of view compared to the RGB cameras (something that occurred on the Kinect 1 and 2 back in the day).
For those that want to relive the moment it was finally announced, the Camera "Passthrough API" was announced today, September 25th, 2024 and is coming in early 2025.
At around 1:56:30: https://www.facebook.com/MetaforDevelopers/videos/449444780818091
"This will enable all kinds of cutting edge MR experiences. You got tracked objects, AI applications, fancy overlays, scene understanding, and so much more".
Let's see what we will actually be given access, since not much was disclosed aside from this phrase about it. All of the things mentioned could be done using different AI models Meta has shown and not only via direct camera access, so hopefully they actually give is direct/raw RGB camera access and access to the depth camera/map as well.
I haven't checked in detail, but I wonder if we will learn more about this API in one of the other smaller talks/developer sessions.To jump right to the camera passthrough API part of the video
https://www.facebook.com/watch/?v=449444780818091&t=7085
It seems I can start updating my quest again, I was really holding the FW for any possible exploit that would be used to root the device and unlock basic features.
Thank you Meta !!
We finally have a Passthrough API coming 😄
Announced at Meta Connect 2024After all this time, is now time to say thank you to Meta!
I also want to thank all the people who supported this post. I am sure this post had to do a lot in putting preassure into them to release the new Passthrough API that's gonna be out next year.
We'll continue posting the news regarding this API in this post.
Thank you you all! Now is time to see if we are actually getting the access we want. If not, let's see each other in a new post.
Wow! This is so exciting! Thanks everyone who helped support getting this out there! Lets go build some cool stuff!
Long overdue, but yes. It's happening. No surprise really that it was announced now, since they had to fix that with the AVP as the competitor. Let's see how much access we are really getting.
This is good metalex201 , getting closer.
I just saw this article about another way to get camera access which might help: https://www.uploadvr.com/quest-3-raw-camera-access-better-workaround/The app makes use of Android's Media projection (https://developer.android.com/media/grow/media-projection?ref=uploadvr.com) and the demo shows the use of the AprilTag as an AR marker to place a virtual object in relation to it and then manipulate its pose (move the marker around).
The project's GitHub Repo: https://github.com/trev3d/QuestDisplayAccessDemo?ref=uploadvr.comI didn't know media projection worked on quest, thanks for mentioning that It will be usefull, althought I'm probably gonna try through ADB to get virtualdisplay working so I can avoid the issues that were listed on github, like media projection needing to be restarted.
I'm so close, yet so far 😧
this is from the package named "qrscanner" which is responsible for the UI and function of the qr scanner used at startup and for wifi code scanning. This is specifically from the library named "libqrcodereader_jni.so"
The app itself uses a class called XrSessionBase, which if I'm correct then it somehow runs an empty VR app in a "virtualDisplay" and uses it as a surface that is then called by this "libqrcodereader_jni.so"
sadly I couldnt get this to work on a decompiled version of the app, because "libqrcodereader_jni.so" is a system library, which is not accessible, So I'm gonna try something else, maybe create a virtualDisplay that just has an empty unity XR app whit passthrough turned on, and if i can somehow get that to run under an application then i could just get the surface from that virtualdisplay and I could use that as basically a raw passthrough layer for an open API, althought thats another roadblock since I have never used unity for AR, and I don't even know how I would run android code from there ._.
