"We built Passthrough API with privacy in mind. Apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors. This means raw images from device sensors are processed on-device."
That's all well and good, but what if an application has a legitimate reason to need access to that data? I can think of several such use cases:
A camera app for taking pictures/recording video in 3D
Mixed reality experiences that implement computer vision algorithms to react to things in the real world
A more advanced version of Waltz of theWizard's mixed reality sandbox that allows changes to be made to objects in the real world (well, at least how they're displayed in the headset 😉) using image processing filters
Trippy effects that go beyond the pre-existing filters in the SDK
I'm sure there are many others as well.
From my understanding, none of these use cases are currently possible on the Quest, even though there's no reason they shouldn't be with the hardware that's present. I couldn't even code it for my own experimentation unless I find some way to root the headset, which is annoying as I spent $1,500 on a Quest Pro and shouldn't be prevented from using its hardware as I please.
So what I recommend is adding an API for accessing this raw image data, which would of course only work if the user grants permission to the app. Apps can already do this on smartphones with no issues, so I don't see how the Quest would be any different. The Oculus Store, of course, can also set policies as necessary, such as requiring apps to keep the image data on device whenever possible, and to never send it to a remote server without the user explicitly giving permission.
I'd have posted this in the SDK Feedback section, but for some reason it says I'm not allowed to start a thread there, so I'm posting it here instead.
I totally agree with that. Please make it happen Meta (at least for the pro headset for enterprise usage). MR application is very limited without the ability to recognize objects in user's surrounding.
Yes we need acces to feed from cameras. How can I build an app similar to remote assist apps if I can't get video stream and send it to expert?! Smth like MS did with Hololens and Dynamic 365 Remote Assist
I agree, as a developer who wants to run my own computer vision code, I don't see why this roadblock is imposed. If it's purely a security concern, well the passthrough cameras are already viewing my room to perform SLAM on device, the microphone is recording, but asks for permission, why can't the camera's do the same.
Sure its nice to have passthrough via OpenXR, but just compositing content onto a camera feed is pretty useless. As per above, this is putting serious roadblocks in place for AR/Mixed reality development.
It's hard to believe what I am reading. This is crazy. How are we supposed to be able to make XR apps without being able to process data about the environment the device is in?
What possible reason could there be for not allowing users to opt in?
Abuse of privacy, and leak of RAW images. Now sure how anyone is somehow missing the big elephant in the room; you can't just dimiss the privacy ramifications with a 'Allow Camera Access" opt in boilerplate message.....
