I want to make an application where one of the steps is to make sense of the items around me, I initially wanted to use the self contained scene understanding but I found out that it has to do a spatial scan first which doesn't meet my expectations, I want him to be real time so I want to get the camera information to use to implement the item recognition function myself.
How to get data from quest3 color camera, or get data from passthrough, but in video streaming form?
If not, is there an appropriate solution for my need?
Sure. In our case, we only need to track a small amount of objects (5) in the app, that is why we can use external trackers. We are using HTC Vive Ultimate Trackers for it. Of course, if in your case you want it to be dynamic tracking for all the environment then this option is not very suitable. For you, the only way of doing it I see is adding a camera on top of the Quest and using that image to run a object recognition algorithm in it. The problem of this is that you have to find the way of passing the information from this camera to a server (to run the algo.) and then pass this info to the Quest.
It's really unfortunate that we have to work in this difficult solutions when there is existing technology to do all of this in the HMD in real time. Good luck with that!
I´m sorry to tell you that right now there is no possibility of accessing the raw image of camera in Quest 3. There is a lot researchers/developers that would use it and would make a lot of projects much easier. However, Meta doesn´t give you access to this for "security" reasons.
As a alternative, we are using external trackers for it, but I am not sure if this will help in your case.
Sure. In our case, we only need to track a small amount of objects (5) in the app, that is why we can use external trackers. We are using HTC Vive Ultimate Trackers for it. Of course, if in your case you want it to be dynamic tracking for all the environment then this option is not very suitable. For you, the only way of doing it I see is adding a camera on top of the Quest and using that image to run a object recognition algorithm in it. The problem of this is that you have to find the way of passing the information from this camera to a server (to run the algo.) and then pass this info to the Quest.
It's really unfortunate that we have to work in this difficult solutions when there is existing technology to do all of this in the HMD in real time. Good luck with that!
