Get world coordinates from screen coordinates

"rkkonrad" wrote:
So by pointing device could I still use a mouse? I will have to read up on ray casting in that case.
The point of my application is to simulate retinal blurring when rendering. I need to grab the depth of the point that a person is looking at (simulated by a mouse click) and apply an appropriate blur to the scene. So I guess what I am really looking for is the depth of a point in the scene. Would ray casting be the way to approach this?

Yes, pretty much. You can get the depth of the point a person is looking (or rather, the point directly in front of them, regardless of where their eyes are pointing) by simply taking a Z axis vector and composing it with the head pose. This will give you the vector to use to find what scene object is being looked at.

Alternatively you could solve this by doing an extra render of depth only information and using client side distortion. Essentially you'd use the SDK distortion as normal when rendering, but when the user clicks a mouse, you could do an additional 'depth only' render, perform the distortion on the depth texture (to a framebuffer), and then read the depth value back out of the framebuffer.

Neither approach is quick and easy. With the raycasting approach you need to find a way to represent the current ray in the scene so the user can manipulate it with the mouse, and you need a way to iterate efficiently over your scene objects in order to find out the closest intersecting object, the intersection point, and the exact depth.