How can I get a depth map(or point cloud) just from my quest 3's depth sensor???

If you don't need a real-time point cloud, and you just want a point cloud from the scan that your Quest 3 did when it did it's room scan, you can use the OVRSceneManager component to return a GlobalMesh prefab that contains a triangle mesh of your space. You can copy the vertices array and just remove the duplicate vertices. If that sounds like what you need, and your need more info on OVRSceneManager and retrieving scene data, let me know and I can go into more detail
- or -
this link may give you what you need to get started: 
Scene Mesh | Oculus Developers

Is there any way to get access to the depth stream (On quest 2/3/pro) ? I am trying to align the headset pose with the pose of a robot (in ROS) to create a digital twin overlaid in mixed reality. One way would be to match the point clouds. Alternatively if the headset is able to detect an April Tag or track a moving object that would work too.

You can access it in the shaders provided with the recently released DepthAPI. If you look at line 28 in the EnvironmentOcclusion.cginc file the environmentDepth variable that is calculated is the raycast depth value picked up by the sensor.

Anyhow it's there, so you can write a compute shader that either directly uses the data, or exposes it to a script.

It's going to take some effort, but it's there.

Hello,

I'm not sure it is what you're expecting to atteign, but I managed to get image and UI occlusion just by activation the Oculus embedded occlusion feature from the Unreal OcclusionSample.
You need, however, to activate scene feature support in the MetaXR plugin settings.

Interesting way of doing it.

I am currently looking into the information of the point cloud data, simply because the placement of the shared anchors sometimes doesnt work and I'd love to know why, hence visualizing / testing the quality of the point cloud.

Do you know JeffNik if the Pointcloud data also stores 2d-color information and shares this? Not for us to access (patiently waiting for next years camera access), but for the fact, that 2d images / logos on a table could help optimizing the point cloud data that is being shared for the shared anchors.

room.png

Update - I just played around with the MRUK. Following the Meta documentation, I found that this new method can be used to get the same room data. When you include the MRUK prefab and EffectMesh prefab in your scene, and do a few other setup items, it creates the same runtime object, representing the geometry it has found or you have added, including the GLOBAL_MESH object containing the scanned geometry to use for a basic point cloud. It looks like this will be the way I do it going forward.

Sorry - having trouble replicating my results - I'll respond after I figure out what I'm doing differently.

Hello,
im pretty new to working with OVR, the Quest 3 and the Depth API but I'm currently trying to generate a live point cloud using Unity's VFX Graph. I want to reconstruct points from the live depth information.
Up until now I know that the Depth API provides the depthMaps, something called the environmentDepthZBufferParams, which hold a depthOffset and invDepthFactor, and the reprojectionMatrices. But I don't really know what a 'reprojection' matrix is, I only know of projection matrices. 
I believe a live point cloud should be possible, since the raw camera data is used to construct the available data. But I wasn't yet able to use this data to reconstruct a live point cloud yet. 
I don't know if this information is helpful in any way but maybe someone has an idea how to use it.

I haven't yet. I think I started to check it out and it came down to having to write shaders, and I wasn't ready to tackle that yet. 😉

Ok, so I figured some things out. 

First of all, it seams that the depth API doesn't actually use the depth sensor but rather computes the provided depth from the two main cameras. I don't think that actually makes any difference but it's interesting to know.
Second, I did it! I was able to understand what the accessible data is and how to create points from that data.

If you want me to I can go into detail as to what the provided data is used for, but for now I will just share how I managed to create a live point cloud from the depth API data.
So the 3D-reconstruction itself is very easy and does not require any special things. You can create a point from the raw data and just need to transform it a little, then multiply it with the inverse of one of the provided reprojectionMatricies and do a final transformation to receive a WorldSpace point.

First of all the data in the depth texture is in a [0,1] space in the R-Channel. From that depth you can construct a 3D point using the UV coords from sampling the texture as the X and Y component getting you a Vector3(U,V,Depth). This point is in ScreenSpace [0,1] and needs to be transformed into ClipSpace [-1,1]. You can do this by multiplying it by two and subtracting one. To get the ClipSpace point into WorldSpace you use the inverse of the reprojectionMatrix. That is basicly the projectionViewMatrix so the inverse transforms points from ClipsSpace into WorldSpace. To be able to correctly transform points though, they need to be homogeneous. That can be acchieved by adding a one to the point making it a Vector4(X,Y,Z,1). Then multiply this point with the matrix. That gets you a homogeneous position as a Vector4 in WorldSpace. At last you create a Vector3 from the X,Y and Z and divide it by the W component. 
And that is it, you now have a Vector3 point in WorldSpace from a sampled pixel of the depth texture.

TLDR: you need to make a point from U,V and Depth, transform it into ClipsSpace, make it homogeneous, transform it into homogeneous WorldSpace using the inverse reprojectionMatrix and apply the perspective divide to the result.

It's important to note that for this pipeline to work the code can't be executed in a normal rendering shader but needs to be executed in like a script, compute shader or similar. For sampling the WorldSpace position inside a normal shader, you also need to transform the U,V and Depth data from the shaders perspective into the perspective the depth is encoded with.