Forum Discussion
When will we get object and image classification (Computer Vision) for Quest 3 and Quest Pro?
If I wanted to build a Mixed Reality app that can detect when a certain brand logo is visible on a poster, coffee cup coaster, etc... and then allow spatial anchoring relative to that logo there seem...
I agree, having access to the raw camera feed is very important for industrial/non-regular consumer focused apllications. If this could be accessed on a Quest 3 or Pro, even if we would have to pay some more for an enterprise version like with the Pico, this would open up the device to many more use cases. Robotics, education, medical, and many other fields would benefit a lot from this due to image processing and object recognition.
Because of the higher field of view (though the warping on the passthrough/video see-through could be improved), I could see this easily replacing development on the HoloLens or Magic Leap if raw video feed, and possibly point cloud data from the depth camera, were to be accessible.
I can only assume this isn't currently available for developers because of privacy issues for regular consumers, but there should be a way to allow for this for enterprise use. Hopefully this will come in the future or else it is a missed opportunity.
Camera feed access is unavailable for privacy issues... Apple is also disabling raw camera feed access on the Vision Pro, but Apple does offer a pipeline where, through ARKit developers can add images and I believe objects as well for recondition. So Apple host an instance of the computer vision model and developers can train that model and have the ability to recognize images/objects and set spatial anchors on those images/objects in real-time... all of this without access to the raw camera feed on Vision Pro. Apple does this without a fee. Apple developers do have to pay an annual fee of around $100 to be part of the Apple Developer program (one time annual fee, which covers all Apple devices), but there is nothing additional and no run-time expenses involved in using ARKit.
See:
https://developer.apple.com/documentation/visionos/tracking-images-in-3d-space
https://developer.apple.com/documentation/arkit/imageanchor
All Meta needs to do is find their own way of providing a counterpart to what Apple is providing through that API to open the door to a large range of MR/AR experiences. Quest 3 hardware is capable, just limited by software in this case. Developers need to be able to detect and perform spatial anchors at least on images detected in the environment.
Meta has invested heavily in AI technologies, including computer vision. They have all the technology to build a closed pipeline without exposing the raw camera feed. You can only build a very limited range of AR/MR experiences without this.
It would be a shame if Meta does make this closed pipeline exclusive to some Enterprise package, that would be closing the door on consumer apps being built that could benefit from this pipeline. Also I think it would be a mistake for them to try to commoditize the pipeline. If they do go that route, I hope they provide a gracious free tier so developers (including App Lab developers) can experiment and release great experiences.
It really makes no sense to allow Apple's Vision Pro developers to build this entire range of apps that can't also be ported to Quest 3, but until Meta builds this closed CV pipeline or allows camera feed access, it really limits the Quest 3's MR/AR use cases to Meta's in-house apps and games that can only build spatial anchors to attach to and augment over basic room geometry and fixtures. It's one the key differences in the device being a toy/console and a spatial computer.