Forum Discussion
When will we get object and image classification (Computer Vision) for Quest 3 and Quest Pro?
If I wanted to build a Mixed Reality app that can detect when a certain brand logo is visible on a poster, coffee cup coaster, etc... and then allow spatial anchoring relative to that logo there seem...
ARKit can for iOS, but from the documentation it appears Apple has only enabled the computer vision model to be trained by developers on images for RealityOS.
Image Detection means it can only recognize some 2D pictures but can not recognize 3D object right? And will apple open its access to object detection in the near future. I truly agree with your points that about the limitations without these algorithms.
You have to realize how computer vision models are trained. There are different approaches to "object recondition" over just identifying an image, but the most common approach doesn't actually involve feeding depth sensor/geometry data to the model. Instead it works something like this, let's say the object you want to identify is coffee mug, a deep learning framework like TensorFlow, Keras or PyTorch might train 2000 images of coffee mugs taken at different perspectives and then it's classifier will do a reasonable job at identifying a coffee mug
In order to do this depth of Computer Vision you really need access to the live camera feed from the device, which neither Apple nor Meta give you.What Apple is offering is a simplified version of image recondition, which is still super useful for many AR scenarios. The developer has to supply all the images to be recognized in the assets folder and they become part of the build, it's really setup for developer to supply one image for each item to be recognized, then programatically you can make Swift augment augment (place 3D rendering) on some point relative to the X,Y offset of the top, left corner of the image.... actually a cartesian coordinate system might be used, I'm not sure, but you can easily render like a 3D model of text showing the price of baseball card or something.
Meta doesn't even give us that and the sad thing is Meta has ahead of Apple on the AI front, this should be deeply integrated.
For anyone interested in this feature,... I am looking at you, Meta developers 😉, see here: https://developer.apple.com/documentation/visionos/tracking-images-in-3d-space
Anyway, I would love to see real object detection for Meta OS, too. And I totally agree, Meta is leading in AI. Why not implement something e.g. CNN based as YOLO?
