Skip to main content
Up and Coming
June 15, 2013

Unified Head and Hand Oculus Positional Tracking

First off, let me say I have no idea HOW this could. I just have an idea that someone should figure it out!

Oculus is working hard on positional tracking for the head, but should they be focusing on hands at the same time? Right now there are all of these different ideas for controllers/gloves/motion tracking. What if you could use any controller or device you want and STILL have positional tracking for your hands that are ALSO relative to your head.

I got to try the Rift with the Hydra, and one frustrating part of that experience was that my virtual hands in the Tuscany demo were not exactly where my real life hands were. I'm sure there are several reasons why this is the case, but the reason I can think of is that the Rift wasn't be tracked, so where my head is located relative to where my hands are located isn't matching up. If the same positional tracking was being used for the Rift and my hands, however, I imagine it would be very easy to match up the hands being exactly where they are supposed to be in relation to my head. But how do you do this and still allow the use of any controller? My idea is positional wristbands.

You could put a wristband on each wrist, making sure that the logo lines up with the center of your wrist. These could be wireless or there could be long wires that plug in directly to the Rift. Assuming that whatever positional tracking solution for the Rift could also work on these bands, you could track the head and both hands in virtual space and in relation to each other. The bands could even track rotation and other movements of the hand. The best part is that this still leaves the hands free to use the input device of you/the games' choice.

Using this method, you could aim with an Xbox controller. You could have a game that could combine the Leap Motion and Oculus' positional tracking. You could still use the Hydra. You could use a glove. Or you could just use the bands (if it were, say, a boxing game). It's about creating an Oculus standard - all Rift compatible games can take advantage of positional tracking for head and hands, BUT it still allows the flexibility for a game to choose different input devices (you could even play mouse and keyboard and just fling your hand around for specific commands or to point at menu options).

Perhaps this idea has already been floating around. Or perhaps the next post will tell me why it's not possible. It's just an idea that, in as much daydreaming about the Rift as I do, I hadn't thought of before, so I thought I would share. Thoughts?

    59 replies

    Up and Coming
    June 17, 2013
    Cute!

    I like the idea of wristbands as well. Would be nice if something like a wireless hydra could split the motion tracking off into a wrist band. That way the ergonomic one handed controllers could be optional (to emulate tools and it is nice to still have buttons, triggers, and joysticks). Also, the controllers could maintain 6DOF, so you could optionally put the "wristband" on your ankles and also measure the relative orientation of the wrist and hand.
    (╯°□°)╯︵┻━┻
    Participant
    June 17, 2013
    Here is my bet for the future.

    - If you remember the "leaked" GTC presentation, the consumer version could come with 2 cameras on the rift.
    - If you take a look to job listings on Oculus website, you'll see that the company is looking for a computer vision engineer and an expert in kinematics.
    - Effective computer vision algorithms for body and hand 3D tracking exist.

    So my bet is that Oculus could use an optical solution in order to track hands (at least). And it's a smart solution :
    - no occlusion : if camera can't see your hand it means your eye can't see it, so no problem.
    - since you know the location of the cameras on the rift, you have unified positional tracking.

    Only my bet . ;)
    Up and Coming
    June 17, 2013
    "LaurentMT" wrote:


    So my bet is that Oculus could use an optical solution in order to track hands (at least). And it's a smart solution :
    - no occlusion : if camera can't see your hand it means your eye can't see it, so no problem.
    - since you know the location of the cameras on the rift, you have unified positional tracking.

    Only my bet . ;)


    If you only want to render somebody's hands then not being able to see them isn't a problem. If you actually want to use a person's hands then you have a serious problem. Take almost any sport as an example. You keep your eye on the ball, not your bat or racket.

    Don't get me wrong, I think head mounted cameras are going to be really useful, but they will need to be integrated with full body cameras and/or sensors.
    (╯°□°)╯︵┻━┻
    Participant
    June 17, 2013
    "jwilkins" wrote:
    If you only want to render somebody's hands then not being able to see them isn't a problem. If you actually want to use a person's hands then you have a serious problem. Take almost any sport as an example. You keep your eye on the ball, not your bat or racket.

    Don't get me wrong, I think head mounted cameras are going to be really useful, but they will need to be integrated with full body cameras and/or sensors.


    I totally agree with you. This would be only a part of a complete solution. Moreover, I don't think it's possible to do full body tracking (which should be an important feature with head and hand tracking) with cams on the headset. We'll need additional cams or sensors to do that.

    But even if it's not really usefull for gaming, I think that having fingers tracking should be a real factor in order to increase immersion. I've done some tests of avateering and it's a bit frustrating to be able to see your hands but to notice at the same time that they're are just like inanimated objects. Well, may be this is my personal professional bias with tactile systems ;) but I'm convinced that hands are really special for humans since they are our primary tool to act on the world.
    Up and Coming
    June 17, 2013
    "LaurentMT" wrote:

    But even if it's not really usefull for gaming, I think that having fingers tracking should be a real factor in order to increase immersion. I've done some tests of avateering and it's a bit frustrating to be able to see your hands but to notice at the same time that they're are just like inanimated objects. Well, may be this is my personal professional bias with tactile systems ;) but I'm convinced that hands are really special for humans since they are our primary tool to act on the world.


    As soon as I can I'm going to be testing mounting the LEAP onto the Rift front plate. I am just waiting to remember to pick up some velcro when I'm shopping next time so that I can make a nice mounting system.

    I've been thinking it might work well for HUD interaction, since the HUD will float with your view and require you to bring your hands into view of the LEAP in order to interact.
    (╯°□°)╯︵┻━┻
    Participant
    June 17, 2013
    "jwilkins" wrote:
    As soon as I can I'm going to be testing mounting the LEAP onto the Rift front plate. I am just waiting to remember to pick up some velcro when I'm shopping next time so that I can make a nice mounting system.
    I've been thinking it might work well for HUD interaction, since the HUD will float with your view and require you to bring your hands into view of the LEAP in order to interact.


    Till now, I've only investigated some tracking with a kinect (I have just received my rift a few minutes ago, I'm going to check if kinect + rift is a good mix) but I'm also interested by possibilities with a sensor like leap motion. I'll follow your results with great interest !
    Up and Coming
    June 17, 2013
    I think a device similar in design to the Tone LG pictured below could help with some neck modeling and lean tracking, which would satisfy a lot of consumer requirements:

    Up and Coming
    June 17, 2013
    "LaurentMT" wrote:

    Till now, I've only investigated some tracking with a kinect (I have just received my rift a few minutes ago, I'm going to check if kinect + rift is a good mix) but I'm also interested by possibilities with a sensor like leap motion. I'll follow your results with great interest !


    This would be interesting because the Kinect uses structured light, but I've never got confirmation on what the LEAP Motion uses. Two structured light patterns overlapping would probably not work well, but neither could it be good for the the 6 IR LEDs in the LEAP Motion to be shining into the Kinect.

    So, I'm really curious if these two devices can get along.
    (╯°□°)╯︵┻━┻
    Participant
    June 18, 2013
    I think a combined Rift & Hydra is pretty close to the way to go. A unified base could receive the video and usb and contain the magnetic orb. One wire goes goes to the headset, and then two wires come down from the headset, one to each wand. Ideally, there would be 3 magnetic sensors total, head and two hands, but I would settle for just head and active hand.

    Regarding dedicated controller vs wristbands or other neutral solution, I don't think you actually need position tracking on your hands while using an xbox controller. I don't see what it would accomplish, but maybe I'm missing something.

    Optical tracking is still a possibility, though. I don't think you need a 360 degree solution, since for now the Rift is wired, so occlusion may not be a big deal.

    Cameras on the Rift itself could be interesting, if they can get marker-less position tracking to work. Then you'd get position tracking and see your hands (and maybe even body) all in one shot, plus you have an easy toggle to see the real world. However, position tracking what's in your hand while it's out of view still remains an issue.
    TboneAuthor
    Up and Coming
    June 18, 2013
    "eshan" wrote:
    I think a combined Rift & Hydra is pretty close to the way to go. A unified base could receive the video and usb and contain the magnetic orb. One wire goes goes to the headset, and then two wires come down from the headset, one to each wand. Ideally, there would be 3 magnetic sensors total, head and two hands, but I would settle for just head and active hand.

    One issue is that Hydra's current solution has way too much latency, not only for the Rift, but for the controllers as well. I don't know why that's the case. Can the magnetic sensors be tweaked? Or is that system just flawed by design?

    Regarding dedicated controller vs wristbands or other neutral solution, I don't think you actually need position tracking on your hands while using an xbox controller. I don't see what it would accomplish, but maybe I'm missing something.

    You're right in that if you want to track your hands while using a controller, you would need a controller that is split in two, like the Hydra. I'm all for a unified Oculus controller with positional tracking - I just figured wristbands might open it up for finger tracking from a different system, or a game that doesn't require buttons to be hands-free.

    Optical tracking is still a possibility, though. I don't think you need a 360 degree solution, since for now the Rift is wired, so occlusion may not be a big deal.

    Having tried out the Rift a few times now, I can tell you that one of my pet peeves is still having to use the right analog stick to rotate left and right. It's a major nausea-inducer, and I'm not sure that any amount of resolution and positional tracking will fix that. The main solution is to make sure the Rift is 360 degrees compatible, so that all turning can be done naturally. For this reason, I can't see optical tracking being a valid solution - it just creates more issues than it eliminates. Get a chair that spins now!

    Cameras on the Rift itself could be interesting, if they can get marker-less position tracking to work. Then you'd get position tracking and see your hands (and maybe even body) all in one shot, plus you have an easy toggle to see the real world. However, position tracking what's in your hand while it's out of view still remains an issue.

    Yeah, based on the leaked consumer prototype picture, it seems Oculus is looking into this direction. You want to be able to do things like have dual pistols and shoot different directions, though. Or no-look shoot around corners. At the very least, though, it might be a cool way to get your body into the game, though it would only work for your POV, so online/multiplayer wouldn't benefit from it.
    The Furious Angels - an Oculus Rift Online Gaming Community!