OK,... Apple did it with VisionOS. An Enterprise API with camera access has been released: https://developer.apple.com/documentation/visionOS/building-spatial-experiences-for-business-apps-with-enterprise-apis
The list of features is wild,... video capture, neural nets for machine learning tasks and object detection with parameter adjustment. As implied by the API name the features are for business applications only, but the first step has been taken! 🙂
I don't want to give a lot of discussion to Apple Vision Pro on a Meta Quest Developer forum, but want to point out that the new Enterprise API for Vision Pro comes with some strong caveats, the big one being you can not build a Vision Pro app with these APIs and list it on the Vision Pro store for general consumer use.
The only Vision Pro hardware that will be able to run apps created with these Enterprise APIs will be Vision Pro devices with the correct security provisioning. I will give kudos to Apple for making this provisioning process easy and the ability to build Enterprise apps doesn't require any special fees to Apple, you're $100/year dev account required to deploy builds on any Apple hardware also gives you privilege to build Enterprise apps and provision them for Enterprise clients.
But this is still a huge feature, it means Apple Vision Pro developers can now port any HoloLens or Magic Leap app, it means Enterprise AR frameworks like Vuforia will now be able to fully support Apple Vision Pro.
Meta has an opportunity to implement this and go further. Yes, there is a privacy concern with giving devs full camera access, no one is arguing that. However, Meta is much further along in the AI space than Apple in many regards. Meta can build a secure pipeline from the Quest's camera feed to their backend models and allow the APIs to just broker this, meaning the image data/video feed flows from cameras to Meta's servers to be processes by Meta controlled AI models and customers can opt-in to running these type of apps without the fear the developer is inappropriately using the camera feed.
It can all be done in a protected way. Meta has the AI technology to do this and gain leverage. Apple is playing catch up with AI, at least on the software side of things.
But will they? Are you listening Boz?
What about on-device image classification/detection, as indicated in the original post? I would suppose that something like e.g. YOLO would run pretty well on Quest 3, Quest Pro or even Quest 2.
Meta did also publish research on Vision Transformers (ViTs) for object detection in 2022: https://ai.meta.com/blog/efficient-accurate-object-detection-for-hundreds-of-uncommon-object-classes/
I feel even the ability to detect a set of standard objects and/or trainable images alone would help to enhance the AR/MR experience significantly.
