Forum Discussion
Passthrough camera data is not available on Quest 3 developing with Unity
Basically it is not possible to access the real time passthrough video from Quest 3 in Unity.
Many devs among the internet think it is a crucial feature to be able to process to create incredible mixed reality apps with object recognition using machine learning. Nevertheless, it is not possible to access the real time video captured by quest 3 in unity.
Here is a reddit post made by me where people are requesting this feature.
Please, we need access to real time passthrough video in Unity or other engines.
109 Replies
+1 on this. Color passthrough access with camera intrinsics is needed for any computer vision-based app development.
I totally agree! Thanks for the support
+1 again. Running CV on the camera feed would allow so many cool applications. World-anchoring of AR content on some kind of marker for example
Honestly, I'm surprised both Apple Vision Pro and now Quest 3 have this restriction. There are so many enterprise-level applications that rely on computer vision. I don't understand how the industry is supposed to start using these headsets if you can't even read QR codes
You are completely right!
They justify the lack of access to passthrough image with "privacy purposes" but it doesn't make any sense. There's no difference between a smartphone app using the camera and a quest 3 app using the passthrough camera.
They can always ask the user for allowance to use the camera anyways.
which was my use case...so I can delegate authentication to a qr code displayed on a PC instead of a crazy chicken dance.
"Note: Currently, Quest 3 can only scan Wi-Fi QR codes from the Meta Quest mobile app. Attempts to scan other QR codes will not work at this time."
It can scan... but they wont allow it for now! hopefully this will change in an update
source: www.meta.com/help/quest/articles/getting-started/getting-started-with-quest-3/scan-qr-code/Rather than simply state "I agree", let me just restate the question I posed to /r/oculus so I can share why I think this feature is so important.
I'm looking to build an app as part of a Ph.D. research project in studying Human Visual Processing - and to do this I've written a Unity Shader which performs a DVS filter (Dynamic Visual Sensor, or Event Based Camera Sensor) effect. I'm very familiar with Unity itself, but I've never found a way to actually apply this shader to real-time vision on an MR headset.
This would essentially be like applying a shader to your own vision in MR. I've obtained a Quest 3 Headset and the passthrough is sufficient enough to actually make this project move, unfortunately there seems to be no way to operate on frames at the pixels and image level in order to actually apply an image filter. I'm wondering if anyone has any other creative ideas on how to implement this.
Regardless of the research, you can imagine a simpler scenario where you have a shader that turns an image to Black and White - so now my question is how can I have the App itself simply display the Front Facing Camera to the user while applying this Black and White shader.
As for computer vision research, access to the image arrays themselves is the only true place to start. I'm fully on-board with notifying users of the cases in which their frames are being accessed, whether it's through a constantly visible icon or not.
+1 on this. Color passthrough access with camera intrinsics is needed for any computer vision-based app development.
Completely agree. I am fine with showing the user a pop-up to ask for camera access.
I really don't understand what the problem is in terms of privacy. How is this fundamentally different than holding up a phone and hitting record? If there was a hard forced feature that indicates that passthrough is being recorded to the user in the headset and a record light to alert people in the environment, it seems identical. Opt-in dialogs should work here as well as on a phone.
I think I am right when I say: everyone agree with you.
I hope they hear all this comments and take these ideas into consideration to release passthrough camera access.
If not, I honestly do not find much interest to develop apps for quest 3. In fact I would even fail to see why quest 3 could be useful as a mixed reality device.
exactly - i got excited by the MR capabilities enough to start trying to learn development cause id like some MR tools that don't exist. Only to find out that Meta doesn't appear to want folks to actually develop for the headset, at least based on the state of the docs.
maybe they are afraid that developers can put malwares links to download to the headset. If they just allow to scan a QR code and something triggers inside the app (not connecting to the web) or make a pop up in the start of the game to be cautious about external links, that would be already very helpful. I hope they enable that soon, the possibilites are great
As far as I'm concerned, it is more a matter of privacy as they say in the docs. Nevertheless, as you said, asking for permission to the user should solve the problem.
Thanks for contributing! I hope they read us
I understand that is 100% a matter of privacy. I was developing for the HTC Elite headset a few months ago and was attempting the same project on their device. On their forums, an actual HTC employee stated it was available for a certain amount of time until corporate partners requested increased privacy.
It's very reasonable to do so; I do not want my headset camera to be accessed without permission, however I am sure that Oculus can construct a secure API.
I imagine you could place four framing edges at the corners of the screen to emphasize this frame-usage even more than the red recording dot. I know Meta has some great UX designers - they can communicate this information very clearly in combination with acknowledgements.
+1
I don't understand why this is not implemented, what a bummer!
(I really need this feature for an XR project involving computer vision)
We are on the same train. Looking forward for this feature to develop an AI MX app for my Msc thesis.
Thanks for the support. I would appreciate if you share the post. Have a nice day!
Accessing the passthrough video feed is an essential feature for many Mixed Reality applications. Dear Meta, we really need this!
