Basically it is not possible to access the real time passthrough video from Quest 3 in Unity.
Many devs among the internet think it is a crucial feature to be able to process to create incredible mixed reality apps with object recognition using machine learning. Nevertheless, it is not possible to access the real time video captured by quest 3 in unity.
Here is a reddit post made by me where people are requesting this feature.
Please, we need access to real time passthrough video in Unity or other engines.
The camera for monitoring the surgery is here inevitably part of the endoscope setup, so not using the passthrough camera of the Meta Quest, and hence not affected by the Quest's lack of pixel-level access to the camera view as needed for computer vision based AR. The video feed with AR overlay of the external system is then likely streamed to the Quest instead of to a conventional external display.
It's strange camera be inacessible since there are no proper reason to that, even privacy isn't enought, since can do like modern phones and ask camera permission and shown some type of icon when recording. Some years back android phones aren't able to record the screen and the community did the dirt work with root, nowdays root is pretty useless for normal people since major features that are really usefull the stock phones included into it. For me lock the system for nonsense things like that just made risk in the end of day, the community may try to do what be needed to exploit the device and that may cause users that just want acess to some basic features have an device with security broken due some dumb missing feature, well, that scenario will happen only if the user base increase a lot and meta keep the basic features locked creating a reason for devs find a way to exploit the system, that's why I think this type of thing to be a very bad decision in all ways.
@bektasesref Very unfortunate comment. It is not a matter of "digging more". It is a matter of Meta to provide the developers the necessary tools to make its own product shine, something valuable and with real use case scenarios.
In addition, this "solution" you propose is far from being optimized, because, at the end, it is recording a screen to then process it, and, if not enough, depends on a PC, so not useful for standalone uses.
Hello! Any updates on this? I am a PhD student starting a new project and would need to share the passthrough image between multiple headsets for a collaborative task. I would have guessed this would have been an updated feature by now but I cannot seem to find anything about it. Thanks!
Recently, Meta showed signs of implementing a privacy feature to completely turn off the camera and sensors when the headset is at rest, which I found particularly strange given the fact that there is no access to the sensors to begin with. It's as if Meta doesn't trust its own system to add this feature, so I believe this might be an indication that they are considering granting access to the camera in the future.
In other news, Leap Motion has updated their software from the "Gemini" to the "Hyperion" SDK. Aside from hand-tracking, it now offers fiducial marker (AR marker) tracking and direct camera access (stereo camera and depth sensor).
Since there are some headsets that use Leap Motion, hopefully this pushes Meta to also allow direct access to their camera someday.
Aside from all of the great industry and accessibility uses, there is a lot of gaming potential as well. A "simple one" is using cards for board/card games. Not sure how many here remember the Yu-Gi-Oh like "Eye of Judgement" PS3 game which used AR markers and the PlayStation Eye Camera - https://www.youtube.com/watch?v=pT4iMP03HJ8&ab_channel=Mystic
