Passthrough camera data is not available on Quest 3 developing with Unity

Recently, Meta showed signs of implementing a privacy feature to completely turn off the camera and sensors when the headset is at rest, which I found particularly strange given the fact that there is no access to the sensors to begin with. It's as if Meta doesn't trust its own system to add this feature, so I believe this might be an indication that they are considering granting access to the camera in the future.

In other news, Leap Motion has updated their software from the "Gemini" to the "Hyperion" SDK. Aside from hand-tracking, it now offers fiducial marker (AR marker) tracking and direct camera access (stereo camera and depth sensor).

https://leap2.ultraleap.com/ultraleap-hyperion/

https://www.ultraleap.com/company/news/press-release/hyperion/

Since there are some headsets that use Leap Motion, hopefully this pushes Meta to also allow direct access to their camera someday.

Aside from all of the great industry and accessibility uses, there is a lot of gaming potential as well. A "simple one" is using cards for board/card games. Not sure how many here remember the Yu-Gi-Oh like "Eye of Judgement" PS3 game which used AR markers and the PlayStation Eye Camera - https://www.youtube.com/watch?v=pT4iMP03HJ8&ab_channel=Mystic

As of now, I think the only immediate solution is to add another camera that you can access. An external webcam, sensor, or (worst case) smartphone mounted to the headset.

I'm definitely considering it; I'm just not sure where to start.

Apple just announced the VisionOS 2.0 at WWDC 2024, which now gives enterprise users/apps raw camera access and other similar features (note that enterprise users must first send Apple a request access to access the new API). For the app to get access to the camera feed features, the user must accept the required access (i.e. camera access), similar to how mobile apps today request further access from user devices.

From Apple's website (https://developer.apple.com/visionos/😞

"

Enterprise APIs

New APIs for visionOS grant enhanced sensor access and increased control, so you can create more powerful enterprise solutions and spatial experiences. Access the main camera, spatial barcode and QR code scanning, the Apple Neural Engine, and more."

Video Explaining the new camera access features: https://developer.apple.com/videos/play/wwdc2024/10139/

API Documentation: https://developer.apple.com/documentation/visionOS/building-spatial-experiences-for-business-apps-with-enterprise-apis

This is a great solution for industry uses. Maybe this may come to consumer apps in the future as well, but for now, this is already a great way to get started working on AR applications. A lot of this seems like ideas from the HoloLens, Magic Leap, and Vuzix devices, which is great as they have had some good uses in the past. Hopefully this will push Meta to do something similar for their headsets, even if only for the Quest Pro or a similar industry tailored headset, which may still require an industry request.

Ivan_aa Thanks for the heads up!

I really have to start digging deeper and working with Vision Pro now. We are getting so many requests from industrial customers about XR use cases with Quest 3 but they all rely on having camera access for computer vision tasks. Now we can at least have another option besides HoloLens 2 and Magic Leap 2.

I hope Meta follows Apple on this. Having those controllers and the precision of control that comes with them would make some of those industrial use cases much better than relying on hand tracking.

ginestopo, I agree, it would be great for Meta to give all developers access to similar sensor data/tools, but companies probably want to be cautious and have some examples of good/safe use of camera access to show the general public that it is safe to/worth giving apps access since there will be a lot of useful/great benefits from it. A lot of consumers are probably worried about privacy, which is valid, but unfortunate for us developers. I think this is one of the many reasons why Apple is first giving access to these features to enterprise developers. I'm guessing that research/university access will probably follow, maybe similar to how HoloLens has a Research Mode (https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/research-mode), before they give every developer access to the raw camera feed.

berglte, you're welcome.

Apple released a video on using object tracking with VisionOS, which I believe does not require an Enterprise API, so all developers should have access. It uses AI/ML for tracking, so developers won't get access to the raw camera feed, it's an interesting solution around the privacy issue of raw camera access specifically for 3D object tracking - https://developer.apple.com/videos/play/wwdc2024/10101/

Apple's tracking is relatively simple for now (object needs to be static, rigid, and non-symmetrical), this has been done with other APIs in the past (OpenCV, ARKit, Metaio - bought by Apple, Vuforia, Wikitude, etc.). Hopefully it will improve later to incorporate dynamic and non-rigid/deformable objects.

If anyone is curious on how object detection and tracking works, here are some guides, algorithms, and papers on the subject (you can find many more online as this is a topic that has been studied for many years now, since the 1970s I believe).

1. 3D Object Tracking GitHub - https://github.com/DLR-RM/3DObjectTracking

2. Papers With Code - https://paperswithcode.com/task/3d-object-tracking

3. Microsoft talk from 2012 on how object tracking works for augmented reality - https://www.microsoft.com/en-us/research/video/3d-object-tracking-for-augmented-reality-handling-multiple-objects-motion-blur-and-lack-of-texture/

4. OpenCV and Object Detection and Tracking (there are many more tutorials available):

https://learnopencv.com/the-complete-guide-to-object-tracking-in-computer-vision/

https://pyimagesearch.com/2018/07/30/opencv-object-tracking/

5. Object Tracking in Computer Vision (2024 Guide) - https://viso.ai/deep-learning/object-tracking/

6. Intel RealSense (types of tracking) - https://www.intelrealsense.com/types-of-tracking-overview/
7. Microsoft Kinect Object Detection and Recognition - https://github.com/ahmedfawzyelaraby/3D-object-detection-and-recognition-with-microsoft-kinect-and-deep-neural-networks

Hopefully this is another push for Meta to step up and release some developer access to camera feed or use one of their AI tools to track pre-trained objects.

+1 on this, I wrote many posts about how we should be able to access the cameras with permissions, just like you would on a smart-phone. Could have options for it to be only accessible for apps running on device, or option to allow the data to be sent to a PC over Link/AirLink etc. Or even some enterprise/developer mode which requires some manual steps to access this API and removes the concerns of users opting into this by accident etc.

I don't really see any real excuse now, we have the market flooded with iPhones and Android devices which both have native camera API's and I've never heard of a case where someone has developed an app that blackmailed a user or anything similar using their camera data. 

Meta released a C++ Framework For Computer Vision (CV) And Augmented Reality (AR) Applications, which includes support for multiple devices, including the Quest - https://facebookresearch.github.io/ocean/

Unfortunately, for the Quest it seems to require attaching an external USB camera instead of using the onboard ones: https://facebookresearch.github.io/ocean/docs/demoapps/questapps/externalcamera/#features

Hopefully this is at least a step forward in the direction of developers getting camera access on the Quest, which, according to Boz in a recent interview (July, 2024), will depend on when consumers are ok with the idea of the Quest recording/capturing what the headset's cameras can see (https://www.matthewball.co/all/bozinterview2024). I'm hoping this won't take until the new Quest 4 and Quest Pro 2 are rumored to be released, in 2026 and 2027, respectively (https://www.roadtovr.com/meta-quest-4-quest-pro-2-leak-the-information/). 

If I understood correctly, at least for now this allows developers to work with cameras on the Quest without having to have it fed through the computer, all of the processing can be done on the headset itself. If this is the case, it will help developers on the Quest be ready for when AR is natively and fully allowed on the Quest via the onboard cameras. This way, those developing now can get a head start on things and allows researchers and companies to use it for internal projects (those that won't need to be published on the Meta Quest/Horizon store). Hopefully, when Meta someday allows access to its onboard cameras, they will also allow access to the depth sensor and point cloud data, giving developers more access than what a standard external camera can give. It would be interesting if you can currently use an external camera with a depth sensor (like one of the Intel RealSense cameras with an IR projector), but I don't think it is supported in framework right now, or will be anytime soon (but I hope to be wrong).

Having this Ocean Framework be Open Source also allows Meta to keep improving it based on developer feedback and catch up to Apple's and Google's AR frameworks and those from other companies (Vuforia, Wikitude, 8th Wall, etc.). Therefore, when it is officially launched on the Quest they have a version that works well and is feature-rich when compared to competitors.

I doubt (as it seems to be AI focused), but maybe we will hear some news about this from Mark at SIGGRAPH, taking place at the end of this month: https://s2024.siggraph.org/program/keynote-presentations/

This strikes me as odd.  We need access to real time passthrough video.  I mean I can ask my meta raybans "What am I looking at" why can't I with the Quest?

Would anyone be interested in creating a group chat where we could figure out a loophole for this? I have researched several parts of the quest OS and I think we could get access to the camera sensor data, well atleast just the passthrough image (AKA the stitched togheter image that is from the cameras, that is visible to the user). I have found some semingly usefull info and i think using elevated privileges from ADB whit the use of shizuku (an app for android that lets apps run code on sort-of system level) I think there is a posibility of developing a community made API that would make camera access possible, but I have been trying for days on end to figure this out but I think this would require a group effort. Since I lack some knowledge in how android as a system works and writting system code for it.