Forum Discussion
Suggestion: Allow apps to request permission to access camera feed(s)
The developer blog says:
"We built Passthrough API with privacy in mind. Apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors. This means raw images from device sensors are processed on-device."
That's all well and good, but what if an application has a legitimate reason to need access to that data? I can think of several such use cases:
- A camera app for taking pictures/recording video in 3D
- Mixed reality experiences that implement computer vision algorithms to react to things in the real world
- A more advanced version of Waltz of the Wizard's mixed reality sandbox that allows changes to be made to objects in the real world (well, at least how they're displayed in the headset 😉) using image processing filters
- Trippy effects that go beyond the pre-existing filters in the SDK
- I'm sure there are many others as well.
From my understanding, none of these use cases are currently possible on the Quest, even though there's no reason they shouldn't be with the hardware that's present. I couldn't even code it for my own experimentation unless I find some way to root the headset, which is annoying as I spent $1,500 on a Quest Pro and shouldn't be prevented from using its hardware as I please.
So what I recommend is adding an API for accessing this raw image data, which would of course only work if the user grants permission to the app. Apps can already do this on smartphones with no issues, so I don't see how the Quest would be any different. The Oculus Store, of course, can also set policies as necessary, such as requiring apps to keep the image data on device whenever possible, and to never send it to a remote server without the user explicitly giving permission.
I'd have posted this in the SDK Feedback section, but for some reason it says I'm not allowed to start a thread there, so I'm posting it here instead.
17 Replies
Literally 2% chance this will happen this decade.
Why do you say that?
I totally agree with that. Please make it happen Meta (at least for the pro headset for enterprise usage). MR application is very limited without the ability to recognize objects in user's surrounding.
its almost like they dont want us to develop XR apps its so frustrating. everything is so awkward for no real reason
Yes we need acces to feed from cameras. How can I build an app similar to remote assist apps if I can't get video stream and send it to expert?! Smth like MS did with Hololens and Dynamic 365 Remote Assist
These Privacy Restrictions go too far at limiting the developers!
I agree, there should be an option to allow saving the images!
I agree, as a developer who wants to run my own computer vision code, I don't see why this roadblock is imposed. If it's purely a security concern, well the passthrough cameras are already viewing my room to perform SLAM on device, the microphone is recording, but asks for permission, why can't the camera's do the same.
Sure its nice to have passthrough via OpenXR, but just compositing content onto a camera feed is pretty useless. As per above, this is putting serious roadblocks in place for AR/Mixed reality development.It's hard to believe what I am reading. This is crazy. How are we supposed to be able to make XR apps without being able to process data about the environment the device is in?
What possible reason could there be for not allowing users to opt in?
Abuse of privacy, and leak of RAW images. Now sure how anyone is somehow missing the big elephant in the room; you can't just dimiss the privacy ramifications with a 'Allow Camera Access" opt in boilerplate message.....
That's never been a compelling problem on smartphones. What makes this any different?
I agree, people keep suggesting the issue is privacy, but smartphones (as mentioned) have this exact same issue. Infact my phones camera is in much more 'riskier' situations to be compromised than my Quest which sits in my office most of the day. Ultimately consumers trust Meta with the camera data because it's processed on device, so why cannot this just be the case for apps to ? Ask user permission, explain to user that the app will have access to cameras while the app is launched, if the user has issue with this, they don't use the app.
There has to be a solution. I hope now we are able to access both scene mesh from Q3's depth and the depth data itself, this could provide some glimmer of hope. However I believe the low resolution of the depth data, does provide some privacy guarantees. Further more there is no access to the colour texture map of the scene mesh itself, which is disappointing, as this could be useful for augmentations.
Totally on board with the sentiments shared here. Meta should reconsider the passthrough camera restrictions. Instead of a blanket ban, why not have app-specific permissions? Let users decide.
We risk falling behind other platforms if we can't fully utilize the hardware. There are plenty of use cases where this would be helpful, for instance in education and healthcare. Also to create richer mixed reality experiences.Privacy is crucial, but there's a balance to be found. Let's keep pushing for it!
I want to say the same thing, imagine in the future when Meta comes out with AR glasses with great features and a lightweight design so that more ppl can use it.
This would even be a boost for accessibility, for people who see little or blink, such as highlighting a certain type of product in a color (books, glasses, pencils, whatever) or other functions.But for now, even in the current state it can be done and surely people will be encouraged to do it.
There are many who pay $3000 for an OCR to be read with TTS, it is very sad that it is so expensive.Being able to support the use of the passthrough camera opens up endless possibilities. They could compete against the Apple Vision.
Education and MR experience is very worth too.
Most useful Mixed Reality applications in the real world will require this. It boggles the mind that Meta is not allowing developers to do this - I don't understand how they're pinning hopes on these devices eventually replacing smartphones and then blocking developers from being able to experiment and develop apps to actually allow this to happen.
