Forum Discussion
Passthrough camera data is not available on Quest 3 developing with Unity
Basically it is not possible to access the real time passthrough video from Quest 3 in Unity.
Many devs among the internet think it is a crucial feature to be able to process to create incredible mixed reality apps with object recognition using machine learning. Nevertheless, it is not possible to access the real time video captured by quest 3 in unity.
Here is a reddit post made by me where people are requesting this feature.
Please, we need access to real time passthrough video in Unity or other engines.
109 Replies
As far as I'm aware in the current tech stack this is not possible, as it would require the Quest to transmit its RGB stream off device and back to the development rig. We've asked for similar features for HoloLens and Magic Leap and the result is the same.
Whether it is a technical limitation or a privacy limitation, who can say. Only way to see the RGB view is through casting using either the MDH (meta development hub) or Remote casting. I doubt it will ever be available in editor.
Hi, both HoloLens (1 and 2) and Magic Leap (1 and 2) offer access to the RGB camera and I've used them successfully for computer vision applications.
EDIT: Okay, I'm not sure if the video stream is available. But single frames certainly are and that's what I'd like to have from the Quest 3.
I work on an unity asset HoloVideoChat and app HoloGuide that allows starting a video and voice chat and sharing the camera with remote support staff. They can then place AR elements to guide someone through a repair or similar.
So far the lack of camera access is the only thing that stops me from porting the software to the Quest 3. There is a whole range of similar use-cases that would heavily benefit from access to the camera!
Hi!
To have full access to the RGB stream might not be completely necessary. But when placing 3D Objects (e.g.) in the real world we need to have a method to align these objects with the real world (camera alignment). Otherwise Mixed Reality will always be limited.
For Magic Leap I do not know, but Hololens lets you access the camera stream to detect certain elements, e.g. QR codes and their alignment. That would totally help to solve many issues some of the developers have right now IMO.Thanks for the support. Nevertheless, access to raw passthrough data is essential for any computer vision application, and it makes no sense to avoid the use of such a feature for a device that is XR oriented.
Have a nice day!
Getting access to camera feed with permission from the user is simply a must for a good XR headset. Theres so much potential here, why not let dev's have access to the data? Meta make it happen!
Dear Ryanality . Due to there is a no negligible number of developers supporting this post here and in reddit, I think it is extremely beneficial for the success of the product, if the Meta team could review this case, discuss about the solutions and provide some feedback to the community.
After all, it is us, the developers who can create value by developing apps for Quest, the ones who can increase sells on the product by creating solutions that have a positive impact in society. But we need tools for achieving it. So please, consider raising this post to the development team so we can have access to raw passthrough video data in our developments.
Thank you very much. Have a great day!
This developer seems to have got it working on the Quest Pro, since it seems like it can use image processing to process image/video from the Quest's camera to detect objects - https://youtu.be/bsVRoffcqJA?feature=shared
https://youtu.be/ggLbaDY02qE?feature=shared
Unfortunately there isn't any information on how this was possible.
Having access to the RGB and also to the depth sensor would be great for robotics research (object and edge detection, distance detection, point cloud access, 3D room reconstruction for autonomous navigation, etc.). Hopefully this comes out as developers are able to do this on other platforms like the HoloLens 2 and Lynx-R1 (https://portal.lynx-r.com/documentation/view/video-capture), which even includes OpenCV commands. A Quest 3/Pro alternative to more expensive platforms like the HoloLens and Magic Leap would be great for university and industry research.
Apple Vision Pro also has some interesting camera features for developers, like image tracking, even though you do not have access to the raw camera feed, it at least is something.
https://developer.apple.com/documentation/arkit/arkit_in_visionos
Hopefully these available features from competitor headsets will push Meta to add camera access, or at least image tracking (for now).
I am quite certain that Apple is not going to grant these permissions given their stance on data privacy. In fact, their primary marketing page explicitly states in no uncertain terms that they will not grant access to the cameras or sensors: (https://www.apple.com/apple-vision-pro/)
- "Data from the cameras and sensors is processed at the system level, so individual apps do not need to see your surroundings to enable spatial experiences."
The API lists only a standard set of Computer Vision tasks, which almost all competitor headsets permit (perhaps with the exception of Image tracking - I suspect Apple will do a good job with this, but will only permit keywords of objects that they have trained their own models to recognize.)
I have to say that unfortunately I'm not the type to hold my breath. I give my support for this - but my solution has been to just build my own AR headset for my own purposes. Eventually, I would like to open source this project for those of you who are looking to build similar applications, but likely not until I've progressed further into my dissertation.
Hi myquestg ! thanks for your support. The answer to the video you attached is no. The user in the video is not using a standalone developed app (the biggest hint is the mouse that appears in the video). He instead is streaming the data to the pc and the pc is processing the image by using some custom computer vision app. So no, he neither had access to video data.
This is impractical, because the interesting apps regarding computer vision for quest 3 relies on the standalone capabilities. So we stay with the same problem... we need Meta to provide access to video data.Have a nice day!
Not just Unity: pixel-level access to raw camera frames is blocked on the Quest devices also for Android apps as well as platform-independent web apps. The privacy argument is weak: my own web app aimed at totally blind users does not store or transmit any images or video, as device developers (at Meta) can verify at API level https://www.seeingwithsound.com/webvoice/webvoice.htm No problems with smartphones, but I'd love to enable blind users to use a Quest for training purposes at home. #blind #accessibility
seeingwithsound That's a perfect example of one of endless possibilities of making use of raw data video! Blind people could benefit enormously from this! No doubt that would be a game changer.
+1 to this as well! We need the ability to have computer vision models running alongside quest 3!
As a XR apps developer it would be very useful add some features like image tracking and so on.. I mean, you guys from Meta are always talking about privacy.. why you can't make something like asking permessions for camera data like smartphones? I mean in this way all the "privacy part" would be resolved, right?
Was excited to explore some of the XR capabilities, and wanted to be able to allow a user to sign into our application using a QR Code similar to how discord does for example. Kind of a bummer.
Theres plenty of logical usecases. Even in the meta workrooms you can video chat from VR to a real webcam, now imagine if I could opt in and share my feed of my passthrough camera. Could help someone build their computer in VR/AR space from the comfort of my desktop, or allow someone to take a progress screenshot from their headset if they are doing something in an XR app. Not to mention all the previously stated training/QR code applications in this thread. Really hope we can opt into this somehow.
Edit: If you cant do this in apple vision pro as well (so ive heard, I havent developed much with it yet), meta automatically gets a competitive advantage if they make it possible. Just a thought.Yeah we should
Totally agree. Oculus or Vision Pro should provide access for researchers/developers to the raw camera feed. If they are considering the privacy problem. Just don't let the application be published in the app store.
You are right! In addition, from my point of view as a developer. They are starting to loose credibility, as they are advertising their product as the "Open" alternative and we aren't even able to access the main component of the device (the passthrough camera).
It feels like they are killing their own potential. They are providing tons of AI models and without the feature we are requesting there's no way of combining both to reveal the true potential of XR.
