Skip to main content
xycloud
New Participant
February 14, 2024
Solved

How to get data from quest3 color camera, or get data from passthrough, but in video streaming form?

I want to make an application where one of the steps is to make sense of the items around me, I initially wanted to use the self contained scene understanding but I found out that it has to do a spatial scan first which doesn't meet my expectations, I want him to be real time so I want to get the camera information to use to implement the item recognition function myself.

How to get data from quest3 color camera, or get data from passthrough, but in video streaming form?

If not, is there an appropriate solution for my need?

      This topic has been closed for replies.
      andoniretegiAnswer
      New Participant
      February 15, 2024

      Sure. In our case, we only need to track a small amount of objects (5) in the app, that is why we can use external trackers. We are using HTC Vive Ultimate Trackers for it. Of course, if in your case you want it to be dynamic tracking for all the environment then this option is not very suitable. For you, the only way of doing it I see is adding a camera on top of the Quest and using that image to run a object recognition algorithm in it. The problem of this is that you have to find the way of passing the information from this camera to a server (to run the algo.) and then pass this info to the Quest. 

       

      It's really unfortunate that we have to work in this difficult solutions when there is existing technology to do all of this in the HMD in real time. Good luck with that!

      View in thread

      3 replies

      andoniretegi
      New Participant
      February 14, 2024

      Hi,

      I´m sorry to tell you that right now there is no possibility of accessing the raw image of camera in Quest 3. There is a lot researchers/developers that would use it and would make a lot of projects much easier. However, Meta doesn´t give you access to this for "security" reasons. 

       

      As a alternative, we are using external trackers for it, but I am not sure if this will help in your case.

      xycloudAuthor
      New Participant
      February 14, 2024

      What are external trackers? Can you tell me more about it?

      andoniretegiAnswer
      New Participant
      February 15, 2024

      Sure. In our case, we only need to track a small amount of objects (5) in the app, that is why we can use external trackers. We are using HTC Vive Ultimate Trackers for it. Of course, if in your case you want it to be dynamic tracking for all the environment then this option is not very suitable. For you, the only way of doing it I see is adding a camera on top of the Quest and using that image to run a object recognition algorithm in it. The problem of this is that you have to find the way of passing the information from this camera to a server (to run the algo.) and then pass this info to the Quest. 

       

      It's really unfortunate that we have to work in this difficult solutions when there is existing technology to do all of this in the HMD in real time. Good luck with that!

      Terms & ConditionsAccessibility statement