Passthrough camera data is not available on Quest 3 developing with Unity

[ginestopo]

Basically it is not possible to access the real time passthrough video from Quest 3 in Unity.

Many devs among the internet think it is a crucial feature to be able to process to create incredible mixed reality apps with object recognition using machine learning. Nevertheless, it is not possible to access the real time video captured by quest 3 in unity.

Here is a reddit post made by me where people are requesting this feature.

Please, we need access to real time passthrough video in Unity or other engines.

[Ivan_aa]

In the "Meet Meta Quest 3S" session, at around 3:37:30, the RGB cameras and sensors of the Quest 3 and 3S are discussed.

https://www.facebook.com/MetaforDevelopers/videos/449444780818091

Both headsets have the same RGB cameras (RGB 18 PPD, 4MP) but are in a different arrangement but are said to be similar in passthrough quality. I assume this will also result in similar raw camera access quality between headsets if they do some blending beforehand, but if not this hopefully won't impact the image feed, and consequently the image processing quality too much.

 

Quest 3 has a hardware depth projector (mainly used during room setup to improve detection and stability on plain/texture-less walls) while the 3S does not (it has a flicker sensor light pipe design and flood infrared illuminators, to improve hand tracking in low-light environments). It was mentioned that depth is calculated via the RGB cameras rather than the depth sensor, which is strange. Maybe the decision to use RGB for depth calculation instead of the depth sensor has to do with the depth sensor having a different and lower resolution and field of view compared to the RGB cameras (something that occurred on the Kinect 1 and 2 back in the day).

[yvettemuki]

Finally...! Exciting moment!

[butterfly557]

Wow! This is so exciting! Thanks everyone who helped support getting this out there! Lets go build some cool stuff!

[michael.mainguy]

which was my use case...so I can delegate authentication to a qr code displayed on a PC instead of a crazy chicken dance.

[ey.konic]

Hi all!
Are there any updates on the release of the Passthrough API or have I missed something?

Cheers

[Ivan_aa]

Unfortunately there haven't been any updates on Meta's release of the camera API other than Andrew Bosworth (head of Meta's Reality Labs) confirming on Instagram (February 3rd, 2025) that it is still planned for early 2025.

In other news though, Google has confirmed that their upcoming OS Android XR will allow developers with camera access, which will work similarly to what is currently done on Android where the app requests camera access and the user must allow access. Though there is no confirmed release date for Samsung's Project Moohan headset, which will run on Android XR, it is expected to release this year.

Hopefully Meta releases their camera API soon or else they will be catching up to even more companies on this front, like Pico 4 Enterprise, Apple Vision Pro, the previous professional AR headsets (HoloLens 2, Magic Leap 2, Vuzix, etc.), all of which already allow camera access to mostly enterprise developers, and now Google/Samsung.

[Ivan_aa]

Meta added headset camera permission description to its documentation 3 days ago - https://www.meta.com/help/quest/511837681585017/

It mentions that users must allow camera access permission, which will allow the app to access the camera and what is in front of the headset (not including any virtual objects/content). The document provides some examples of possible camera uses (object tracking, location-based, and using machine learning to process camera feed). It also mentions access to real-time camera feed (which I am assuming and hoping means raw access) and intrinsic camera values (camera focal length, principal point, image dimensions and camera position relative to center of device).

I did not find any mention of camera access in their document on viewing and managing permissions (which was updated 5 days ago), so maybe there are still more things to be updated first before release: https://www.meta.com/help/quest/337925588963509/

At least it looks like the wait won't be too much longer. I'm hoping it comes out before the end of April or else it won't be early 2025 anymore.

Update: Meta will be giving some updates on their camera access at GDC.

[Ivan_aa]

We finally now have camera access on the Quest 3(S) \o/

See documentation: https://developers.meta.com/horizon/documentation/unity/unity-pca-documentation

Sample project/code from Meta: https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples

A video has already been created showing how to access it in Unity: https://youtu.be/1z3pcMJbnRA?feature=shared

Tutorial on getting it working on a new/empty project: https://skarredghost-com.cdn.ampproject.org/c/s/skarredghost.com/2025/03/17/how-to-camera-access-met...

List of projects using camera access created by the community: Here is a list of what some developers have already created with the Quest camera access - https://github.com/xrdevrob/QuestCameraKit?tab=readme-ov-file#community-contributions

Current Limitations

1. No Unity Editor Camera Access: Unfortunately there isn't a way to access the camera via the Unity Editor, which was a useful feature with other AR devices, like the HoloLens, to speed up development.
2. Apps can't be published on the official store: Applications can't be posted on Meta's store for now. Not sure if you can create it for desktop/pcvr though, in case you need to more processing and rendering power.
3. Camera Resolution not Full-HD: The camera resolution isn't great, only 1280 x 960.
4. Low/Mismatched Data Rate: Only 30Hz, much less than the refresh rate of what is being rendered and calculated.
5. No Raw Depth Sensor Access? I'm not sure if we have raw access to the depth sensor, for instance for point cloud generation.

Hopefully these things will be improved in future software and hardware updates.

[dario.maciel]

just commenting here so nobody forget about Unreal devs please!!!
passthrough API for unreal!!

[seeingwithsound]

Same thing with camera web apps. Things that work in modern browsers on all other main platforms (Android phones, iPhones, Windows and Linux PCs, Rasbian devices, etc) ought to also work in the Quest 2 and 3 browser

https://www.seeingwithsound.com/webvoice/webvoice.htm

Us devs have better things to do than jump through platform-dependent quirks and policy restrictions. Privacy excuses in this case do not apply because per default no video frames are stored nor broadcast.