Get world coordinates from screen coordinates

"rkkonrad" wrote:
I am working on a program that requires knowledge of depth information of a particular point on the screen. Currently, I want to get this point through a mouse click on the screen. Because I am not accounting for the distortion and my point in world coordinates is wrong. Is there a method to undo this distortion, so that when I click on the screen I am able to recover the mouse click in coordinates of the undistored image.

The standard answer given by Oculus is not to use the 2D mouse coordinates, but to use a pointing device to project a ray into the scene, and then use that ray to determine what's being hit. Whether or not this is applicable to your situation isn't obvious.

However you should bear in mind that converting 2D screen coordinates into world coordinates isn't necessarily a simple matter of inverting the distortion function. Remember that if you have timewarp active, the image projected on the screen will be both distorted and slightly rotated to account for head movement in the interval between rendering the content and displaying it on the screen. Even if you write the code necessary for doing both inverse transformations, getting the exact time warp matrices used by the most recent frame render might be tricky.