Forum Discussion
(OLD) Passthrough Video for Image Processing (e.g., Real Object Detection/Tracking) is Not Possible
I wanted to share some insights regarding the passthrough API and object tracking for AR projects.
Initially, I thought the passthrough API would provide access to the RGB camera feed for processing with OpenCV. However, it turns out that Meta does not allow access to the RGB feed due to privacy and security concerns. Instead, they handle the processing directly on the Quest device, and applications like Unity or Unreal Engine are not provided access to this video feed.
The only viable solution would be to stream the camera feed live from the Quest 3 and process it on that stream, but there may be unforeseen challenges with this approach, so I decided to abandon this direction. If you're planning an image processing project with Quest, I'd recommend being aware of these limitations.
*sighs out*
EDIT: Passthrough Camera API finally gives permission to access RGB cameras. Link
EDIT: Passthrough Camera API is up for Unity but there is some issues with Unreal. It is possible but there is no official sample(yet?). Link
5 Replies
I understand the privacy and security concern, but don't understand why the access is not given for locally-installed apps. It's fine if they don't want to have apps with camera access in the store, but why not give local projects and art installations and museums the chance to detect an exhibit and then display additional information and a 3d model you can interact with in XR, how cool would that be?!?
Exactly, that's the whole point of MR apps. Some have said elsewhere that they'll bring limited access in early 2025, but for now that stalls development.
I hope that's the case. Could you tell me where you have read that or link a source?
Here you can find a repo that implements the Android Camera2 API in Unreal and with which you can access the headset cameras of the Meta Quest 3 headset (passthrough): https://github.com/yesidfonseca/AndroidCamera2-UnrealEngine/tree/Passthrough_Quest3
