Skip to main content
mka
Up and Coming
November 7, 2014

Re mouse picking

The program I have developed for the Rift (which uses OpenGL) needs the ability to pick objects
with the mouse. What one needs is a map M which maps pixel coordinates (x,y) on the Window or
monitor screen to pixel coordinates (x',y') in the texture buffer (pRenderTargetTexture of
renderDevice). In DK 1 with client distortion rendering, I defined this mapping M myself using the
information we were given about the distortion map. However, the recommended approach
for DK 2 is to let the SDK do the distortion rather than do client distortion rendering. If the distortion
is going to be hidden in the SDK, then it seems to me that the SDK should also supply the map M.

My question therefore is: Does the Oculus development team have plans to do this for the SDK?
(I am not aware of the map M being available currently.)

At the moment I will write the map M for DK 2 myself on the assumption that the mathematics
for the distortion has not changed. But who knows what kind of distortion maps will be used in
future SDK's and it would therefore be inconsistent to recommend SDK distortion rendering while
expecting developers to write the map M themselves.

Hopefully, there is someone from the Oculus development team that can answer this question!

    14 replies

    Show previous replies
    mkaAuthor
    Up and Coming
    November 11, 2014
    With DK 1 I used the information in the DistortionConfig class to compute the appropriate inverse functions. See the Util_Render_Stereo.h in the DK 1 and the StereoConfig class which had a member called "Distortion" of type DistortionConfig (this no longer exists in DK 2). Here are the basic steps I used for my mouse picking:

    1. Windows would give me the position (x,y) of the mouse on the computer screen
    2. I would map (x,y) to its value (x',y') in the texture buffer via the inverse distortion maps
    (The texture buffer is what the Rift DK now maps to the screen via distortion)
    3. My color coded objects would be mapped to the texture buffer
    4. I would check what color the pixel (x',y') had

    To repeat this process in DK 2 I need to know how to compute (x',y') given (x,y), i.e., the inverse of the distortion map.

    At this point I haven't analyzed how the new distortion mesh stuff works. Previously, computing the inverse of the distortion map amounted to finding the inverse of a polynomial map M whose coefficients I was given. If you tell me what that M is, I can do the rest. It would be nice if the DK 2 had a place where it said "This is the formula for M: ... " like there basically was in DK 1.
    mkaAuthor
    Up and Coming
    November 11, 2014
    My main effort the last week or so has been to convert my code so that my program will run under DK 2. I am mostly done except for the mouse stuff which I am only getting to now. My first priority was to get things running without the mouse for now and hopefully the main Rift interface will all work ok when I try it out.

    I was looking some more at the files OVR_Stereo.h/cpp today and I think that they may have, after all, what I need as far as getting the inverse of the new distortion functions that I have been talking about above. I guess I hadn't looked at
    those files carefully enough. It will take a little while to finish implementing everything and running with the new Rift. When I do I will let you know how it all worked out.
    HartLabs
    Up and Coming
    November 11, 2014
    "mka" wrote:
    1. Windows would give me the position (x,y) of the mouse on the computer screen
    2. I would map (x,y) to its value (x',y') in the texture buffer via the inverse distortion maps
    (The texture buffer is what the Rift DK now maps to the screen via distortion)
    3. My color coded objects would be mapped to the texture buffer
    4. I would check what color the pixel (x',y') had

    OK I think I finally have my head around what you are doing and have some suggestions if that Oculus function does not do what you need.

    You could take the mesh and use it to create your own mapping function. It would not be continuous as your current function is. Instead find what triangle the mouse falls in, check the positions of its corners, and generating the mouse position using the associated texture coordinates.

    Alternatively you could possibly render your colored object scene, distort it to a secondary buffer with no aberration and check directly against it.

    Then finally throwing all that out the window, depending on what your program actually is, you could just pick at the center of the screen. Using the physical mouse in VR I find very awkward to begin with, let alone the visual system mouse. Have you thought about a gamepad type controller for buttons head movement for selection? I do that but using a view-able laser cast out of just below center of the dominant eye. It works really well.
    Synthesis - Abstract Molecular Dynamics Simulator @HartLabs
    mkaAuthor
    Up and Coming
    November 12, 2014
    Thanks for your suggestions HartLabs. I'll keep them in mind when I work on my mouse code.
    Terms & ConditionsAccessibility statement