Using Depth Buffer Variables for Collision Detection

I assume you want to be able to compare or sync the depth buffer from the camera, and your virtual world, right?

You should make a physical rig with some objects in front of the camera at known distances from the real camera.

Then make a 3d scene with your objects, using the dimensions you record in the real world test rig. Measure the real world in meters, and make sure the units in your 3d model are also meters.

Then you make a 3d renderer where you can overlay the cameras image on top of your virtual world to compare.
For example if there is a cube in front of the real camera, you should basically see that same view in your virtual world.
You will then adjust your projection matrix until the images align. Basically field of view, and aspect ratio need to be adjusted. Good luck with that part!

Once your view looks correct, you would then make a custom pixelshader that will create a depth buffer. These are easy. The VertShader passes the vertex Z, and you basically scale it and write it to a texture in the PixelShader. You would play with that scale factor until it sort of matches the one generated by the camera.

At this point you have 2 compatible, comparable depth buffers, one for the real world, and one for the virtual world.

Cakewalk. :geek:

I've already done exactly what you described Mid4.
In 2013 I was using DK1 + Creative Senz 3D, then I've switched to DK2 + RealSenz3D.
Here is example application :
https://www.youtube.com/watch?v=QtrPuYeh_NY
(part interesting you from 0:46 but worth watching all).

What interest me, is if you were able to use BOTH RealSenz cameras at the same time.
According to Intel SDK is not supporting that.

"matlock" wrote:
I assume you want to be able to compare or sync the depth buffer from the camera, and your virtual world, right?

You should make a physical rig with some objects in front of the camera at known distances from the real camera.

Then make a 3d scene with your objects, using the dimensions you record in the real world test rig. Measure the real world in meters, and make sure the units in your 3d model are also meters.

Then you make a 3d renderer where you can overlay the cameras image on top of your virtual world to compare.
For example if there is a cube in front of the real camera, you should basically see that same view in your virtual world.
You will then adjust your projection matrix until the images align. Basically field of view, and aspect ratio need to be adjusted. Good luck with that part!

Once your view looks correct, you would then make a custom pixelshader that will create a depth buffer. These are easy. The VertShader passes the vertex Z, and you basically scale it and write it to a texture in the PixelShader. You would play with that scale factor until it sort of matches the one generated by the camera.

At this point you have 2 compatible, comparable depth buffers, one for the real world, and one for the virtual world.

Cakewalk. :geek:

It doesn't work like this at all.

First of all Depth sensor and Color sensor are located in different places, so their frustums don't overlap each other but only intersect sharing part of 3D space. Second thing is the fact that both that sensors have different horizontal and vertical FOV's which means they have different angular coverage in samples (and we can choose from variety of resolutions and refresh rates we want to stream from them as well). Third is the fact that DK has also different FOV's which is much bigger than area sampled by depth sensor. None of them overalapp, you cannot simply blit/merge two depth buffers together. Any you don't want to do it anyway. Another point is that you can query that angles from SDK so you can exactly calculate size of frustum for both sensors in cm in real world. Using that and knowing exact sensor position related to HMD you can precisely reconstruct real objects in VR using sampled depth and color buffers.

Thank you for all your replies!

So to be honest, the part where i need help is a different one than you thought.

I dont know how to get the data of the depth buffer in "TinyRoom" and how to store it correctly.
I am completely new to the depth buffer and never used it before, so this is where i need help.


About all the other things afterwards you dont have to worry. We already planed all of this and made already some stuff.

Thank you all for your help!

While mrkaktus has his foot in his mouth, I will mention again, the easiest way to get the depthbuffer of an opengl program is to make a special shader that writes the z value of the geometry to a texture. These zbuffer shaders are used all the time when making ambient occlusion shaders, and those shaders are all over the web.

"matlock" wrote:
While mrkaktus has his foot in his mouth, I will mention again, the easiest way to get the depthbuffer of an opengl program is to make a special shader that writes the z value of the geometry to a texture. These zbuffer shaders are used all the time when making ambient occlusion shaders, and those shaders are all over the web.

Hmm, enlighten me and please say where I've put my foot in my mouth?

By the way, you're wrong again. It's not an easiest way. There is completely no need to write "special shader" to write Z value to texture. You will get the same result by just binding texture with depth format to depth attachment of FBO. There is also no point in reading "back" depth of a 3D scene, as masking can be done on GPU side (leveraging parallelism), so it is easier and better to send RealSense depth buffer to GPU and just use it for rendering. You can even benefit from Z-Test to do that job partially for you if you will have proper preprocessing done.
But I assume you've already done that smart pants?

@mid4:
Are you able to use both F200 cameras at the same time with RS SDK?
Or are you using some other lower level SDK for that?
According to Intel support, they aren't supporting more than 1 sensors at the same time.