New Participant

Solved

Can I access Quest 3's front camera?

19 replies
19576 views

I want to make MR app using Computer Vision technology.

But I don't know How can I access my VR's front camera.

I want to get front camera's image to process.

Can I access Quest 3's front camera?

This topic has been closed for replies.

Inspiring

You can now access the Meta Quest's Passthrough Camera: https://www.youtube.com/watch?v=1z3pcMJbnRA
See samples here: https://github.com/xrdevrob/QuestCameraKit

Official Passthrough Camera Samples: https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples
Documentation: https://developers.meta.com/horizon/documentation/unity/unity-pca-overview

Honored Contributor

The quick answer is "no". For privacy reasons, you can't directly access the images from the passthrough camera or the depth sensor to use with non-Meta libraries or custom programming. However, with the assumption that you are using Unity - the Meta SDKs offer MR capabilities through the Passthrough API. Basically - you add some components and tell the SDK to render your passthrough behind (or OVER) your VR rendered objects, and Meta does it behind the scenes for you without giving you access to the actual image/video/depth data. Here is a link to the documentation page for the passthrough feature in Unity:

Get Started with Passthrough | Oculus Developers

If you aren't using unity, you should be able to navigate the docs and learn about how to use it in other programming environments. I'm happy to give you more specifics if you still have questions.

Participant

To me this is a very important feature request, hope meta gives us access soon.

New Participant

The privacy argument is not valid.

On android based systems (such as the one used for the Q3), the user has to give his permission for a specific app to use the cameras so this argument flies out of the window.
Now I'd love to know the real reason to why this essential feature is not yet implemented.

Honored Contributor

I don't disagree. As much as I'd love access to the camera/depth data, I've been programming in Unity for Oculus for almost 10 years, and they haven't given us access yet. They feel strongly about it for some reason, and I've pretty much given up hope. 😉

Participant

Can we get a reply from Meta about this? This doesn't any sense to me. I would also like to start working on computer vision applications using the quest 3. That's one reason I bought one...

We have a device that can do spatial tracking. What if I wanted to track against irl objects to power an application? This has many uses.

1. Track specific pieces on a VR tabletop game, showing information/animations on top of them.

2. Ask quest to 'track' an object you are holding and show you where it is afterwards. (universal, 'find this for me' app)

Is there an obvious alternative to these methods? Can we access the display data perhaps during a passthrough and use that for training computer vision models?

Up and Coming

Hello butterfly557 !

The biggest and most supported post regarding this issue is here.

Many people had your same idea and we need even more support in order to Meta hear us.

Thanks and have a nice day!

Participant

Hello ginestopo!

Thank you for pointing me in the right direction! I have replied and 'kudo'ed that post so we can have a louder voice.

Inspiring

You can now access the Meta Quest's Passthrough Camera: https://www.youtube.com/watch?v=1z3pcMJbnRA
See samples here: https://github.com/xrdevrob/QuestCameraKit

Official Passthrough Camera Samples: https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples
Documentation: https://developers.meta.com/horizon/documentation/unity/unity-pca-overview

Inspiring

Great! Now I wait for Unreal support...

Honored Contributor

Fantastic news to see it finally arriving! I may have to wait to play with it though - part of this process is to manually add the permission to the android manifest file. Because of the way that Unity handles the android manifest, I've never been able to find good documentation on how to do this - I add the permission, but it's always removed when building the APK. Anyone know what I might be doing wrong or have a step-by-step guide I can check out? Plus - if the device will only allow one attempt to set the permission (as xrdevbob suggests) - I'm sure I'll mess it up, and have to factory reset a bunch of times. 🙂

Inspiring

Don't worry, the manifest should not change if you don't update it. There is a Meta option to update the Manifest as well as create a store compatible manifest. I would just do this once and then add the camera permission manually. Once you've done this the manifest does not just change by itself.

Also for the camera permissions, this only happens if you deny all the permissions. But even then, all you have to do is to create a new build (incremental is fine). You do not have to factory reset your device.

Feel free to ping again if you get stuck with anything!

Honored Contributor

Sorry to ask, but I've tried those things in the past, and it didn't work for me - my changes kept getting overwritten. Could you give me the actual steps involved?