Forum Discussion

kutlaytunc's avatar
kutlaytunc
Expert Protege
6 months ago

(OLD) Passthrough Video for Image Processing (e.g., Real Object Detection/Tracking) is Not Possible

 

I wanted to share some insights regarding the passthrough API and object tracking for AR projects.

Initially, I thought the passthrough API would provide access to the RGB camera feed for processing with OpenCV. However, it turns out that Meta does not allow access to the RGB feed due to privacy and security concerns. Instead, they handle the processing directly on the Quest device, and applications like Unity or Unreal Engine are not provided access to this video feed.

The only viable solution would be to stream the camera feed live from the Quest 3 and process it on that stream, but there may be unforeseen challenges with this approach, so I decided to abandon this direction. If you're planning an image processing project with Quest, I'd recommend being aware of these limitations.

*sighs out*

EDIT: Passthrough Camera API finally gives permission to access RGB cameras. Link

EDIT: Passthrough Camera API is up for Unity but there is some issues with Unreal. It is possible but there is no official sample(yet?). Link 

4 Replies

  • lucajunge's avatar
    lucajunge
    Explorer
    6 months ago

    I understand the privacy and security concern, but don't understand why the access is not given for locally-installed apps. It's fine if they don't want to have apps with camera access in the store, but why not give local projects and art installations and museums the chance to detect an exhibit and then display additional information and a 3d model you can interact with in XR, how cool would that be?!?

    • kutlaytunc's avatar
      kutlaytunc
      Expert Protege
      6 months ago

      Exactly, that's the whole point of MR apps. Some have said elsewhere that they'll bring limited access in early 2025, but for now that stalls development.

      • lucajunge's avatar
        lucajunge
        Explorer
        6 months ago

        I hope that's the case. Could you tell me where you have read that or link a source?