Skip to main content
Up and Coming
January 13, 2014

Should Oculus bundle motion controllers with every Rift?

Hi there!

I've mentioned this a couple times back around September, but now that it's official that Oculus VR is focusing on a seated experience and that external optical tracking looks likely for CV1 i think it's become relevant again..

Also posted a comment on Reddit the other day, just thought i would create a new thread here to see if we can get a bit more feedback.
http://www.reddit.com/r/oculus/comments/1uw0nl/the_next_big_hurdle_for_vr_getting_your_hands_in

Here's a slightly edited version of my reply to the OP:
"PatimPatam" wrote:

Basically, i believe that the company in the best position to put our hands inside the game is actually Oculus VR itself.

I would love to have an affordable haptic feedback glove, but for the 1st generation of consumer VR i don't think we really need one; a motion controller similar to the PS-Move but where each finger rests in a separate analog button (with optional force feedback) should be enough to simulate our hands. For instance I can imagine it having the shape of a prism or a dumbbell, with several LEDs on both ends.

Having already an external camera for positional tracking this type of controller should be very cheap to produce for Oculus (basically just plastic, a few LEDs, a bluetooth transmitter, and one of the awesome IMUs they already developed to use inside the headset).

I think Oculus could perfectly bundle 2 of these controllers with every Rift sold, without adding more than $40 or $50 to the final price.

I know it may not sound like super-exciting high-tech at first, but the fact that EVERY Rift user would have the same VR input would make a MASSIVE difference. Developers could create VR games and applications with very intuitive interfaces from day 1 knowing that there is a standard input shared by all users.


And here's a very rough sketch i made to have an idea of what i mean. On top of that there should probably be a strap that goes around the back of your hand, so the controller stays in place even with the palm open.



Also optionally there could be a thumbstick instead of the thumb button, but i don't really think it's necessary since i believe avatar movement/rotation could be archived in more intuitive ways, like pressing a button and using positional tracking (lean forward to move forward for instance). Thumbsticks for movement have become quite standard between gamers but i don't think they are that intuitive for the average person.

More on that here:
http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=18694


Obviously the main technical difficulty with this solution would be how to differentiate controller LEDs from HMD LEDs. The more trivial approach would probably be using a visible spectrum camera instead of infrared and have different colors for each controller but that has it's own set of disadvantages.. anyway i think with the team of computer vision experts at Oculus this should be certainly doable.

I'm also quite certain that a system like this (if used at an optimal distance from the camera) could be much more accurate than alternatives like magnetic tracking (mainly due to the magnetic distortion problem).

At the same time systems like the STEM or PrioVR could still be useful for scenarios where the user wants to move around with more freedom within a confined space, or inside omni threadmills.


I've seen quite a few people complaining about the external camera solution.. Am i the only one INSANELY excited about it?? I believe this could potentially bring us one step closer to the Oasis console.

I honestly think this should be a no-brainer for Oculus, and actually i'm betting they already have some hand-controller internal prototypes; then again maybe i'm wrong :-)

Thoughts?


EDIT:

Like i mentioned later in this thread, if occlusion is too troublesome to sort out with only the existing front camera, a 2nd inside-out camera with a wide FOV could be used on the headset to help track the controllers. If the HMD is blocked by the controllers, the relative position between the external camera and the HMD could be calculated integrating the relative position between camera and controllers and the relative position between controllers and HMD.

Maybe this inside-out camera could be used at the same time for pass-through functionality similar to Gear VR. Obviously this would add both weight and cost to the HMD, but i don't see it as a show-stopper.

    132 replies

    Up and Coming
    September 28, 2014
    Here is one of them, found all kind but I like their article: http://www.roadtovr.com/leap-motions-next-gen-dragonfly-sensor-designed-vr-headsets/
    Up and Coming
    September 28, 2014
    yeah the new one looks amazing and I like the color and wide fov of the new one. I do not like that they plan it as an embedded option for VR headsets. I just do not see Oculus licensing the tech and I do not see any other headset getting a share of the market big enough for the tech to build up a userbase big enough to draw developers attention.
    I'd love to be proven wrong though ;)
    check out my Mobile VR Jam 2015 title Guns N' Dragons
    Up and Coming
    September 29, 2014
    Even if it worked perfectly as advertised i'm not convinced about the LEAP Motion because of the fact that you don't have any physical feedback, and simple things like holding a virtual gun would feel very gimmicky. I agree that not having to deal with other periferals would be a big win for mainstream usability, but for me it's not worth the tradeoffs. Still would be better than nothing i guess.

    But in order for it to work properly they would have to reduce a lot the latency from the current LEAP (something that i haven't seen mentioned anywhere on the "Dragonfly" announcement).

    On top of that i'm quite skeptical when they say blocking 3 of the DK2 IR LEDs won't affect negatively the Rift's positional tracking.. It is true that it won't affect it IF those are the only 3 LEDs that you are blocking, but precisely having both hands in front of your face is something that already challenges (and breaks) positional tracking using the current DK2. Blocking an extra 3 LEDs will make it even worse, and i really can't see an easy fix for that.

    Those are 3 important issues that a system similar to the one i proposed would not have.
    Up and Coming
    September 29, 2014
    "lmaceleighton" wrote:
    After talking about this I almost bet our Leap guys chimes in here. I bet he will have more to say, but if not I'll look for the article if anyone is interested.


    My ears are burning...

    "lmaceleighton" wrote:
    Here is one of them, found all kind but I like their article: http://www.roadtovr.com/leap-motions-next-gen-dragonfly-sensor-designed-vr-headsets/


    ... but it looks like you got this!
    Head writer @ Leap Motion
    Up and Coming
    September 29, 2014
    "PatimPatam" wrote:
    Even if it worked perfectly as advertised i'm not convinced about the LEAP Motion because of the fact that you don't have any physical feedback, and simple things like holding a virtual gun would feel very gimmicky. I agree that not having to deal with other periferals would be a big win for mainstream usability, but for me it's not worth the tradeoffs. Still would be better than nothing i guess.


    That's a fair point, and there are a few potential approaches to tactile feedback. One is that touchless motion control enables a broad variety of experiences beyond the traditional point-and-shoot model; another is that compelling visual and audio feedback can create the illusion of tactile response. But for the specific example you gave, our API makes it possible to attach IR markers to real-world objects and make them trackable in 3D space.

    "PatimPatam" wrote:
    But in order for it to work properly they would have to reduce a lot the latency from the current LEAP (something that i haven't seen mentioned anywhere on the "Dragonfly" announcement).


    Reducing latency is extremely important, and any latency that can be registered by the human nervous system is unacceptable. That being said, Leap Motion technology is already capable of cutting under the human perception threshold; otherwise musicians wouldn't be able to use it.

    At the moment, we haven't yet optimized our tracking algorithms for beta HMD-mode tracking. Mostly because they're evolving too fast for optimization to be worthwhile, as we're currently focused on improving other forms of tracking performance (e.g. more robust recognition of different hand poses). Ultimately, our goal (not just for latency, but for tracking overall) is that, if you're looking at the raw infrared imagery, the Leap Motion Controller's ability to recognize and follow your hands should be equal to (or even better than) your eyes.

    Right now, the latest version of the software involves typical processing times in the 3-7 millisecond range. (Though of course that gets combined with other factors, e.g. display system processing, to create a "sum of all latencies" at the other end.) This will drop in the weeks and months ahead.

    "PatimPatam" wrote:
    On top of that i'm quite skeptical when they say blocking 3 of the DK2 IR LEDs won't affect negatively the Rift's positional tracking.. It is true that it won't affect it IF those are the only 3 LEDs that you are blocking, but precisely having both hands in front of your face is something that already challenges (and breaks) positional tracking using the current DK2. Blocking an extra 3 LEDs will make it even worse, and i really can't see an easy fix for that.

    Those are 3 important issues that a system similar to the one i proposed would not have.


    I definitely appreciate your skepticism. For what it's worth, we haven't experienced any issues over months of development and experimentation, and I haven't heard of any devs running into problems since we released the first VR build last month.

    It's also worth mentioning that at this stage, both technologies are experimental -- the headset itself, and using the Leap desktop peripheral in this way. We're working on Dragonfly precisely because we believe that a module that VR OEMs can embed into their devices will be the best way to ensure a seamless VR/AR experience.
    Head writer @ Leap Motion
    Up and Coming
    September 29, 2014
    I would be really interested in IR Marker! I ask on the Leap Forums for help getting another type of tool to be recognized, and while he said that the tool I wanted to include would not work because it was too think(Tools need to be like a chopstick I guess)...anyway it would have been a more complete answer for him to include that I can track IR markers with the Leap. I would love to have some information on this. Can you point to some examples, as this is a major part of our game,and is something I was using Open Source Tracking solution with my PSeye CAM. Iy would be nice if it was an all in one solution. Essentially I need to track a Fully Auto Gun Prop....(Did I just break NDA?...guess Ill find out).

    EDIT: Found this one :https://community.leapmotion.com/t/optional-gloves-with-ir-markers-for-improved-accurancy/1726
    Up and Coming
    September 29, 2014
    "lmaceleighton" wrote:
    I would be really interested in IR Marker! I ask on the Leap Forums for help getting another type of tool to be recognized, and while he said that the tool I wanted to include would not work because it was too think(Tools need to be like a chopstick I guess)...anyway it would have been a more complete answer for him to include that I can track IR markers with the Leap. I would love to have some information on this. Can you point to some examples, as this is a major part of our game,and is something I was using Open Source Tracking solution with my PSeye CAM. Iy would be nice if it was an all in one solution. Essentially I need to track a Fully Auto Gun Prop....(Did I just break NDA?...guess Ill find out).

    EDIT: Found this one :https://community.leapmotion.com/t/optional-gloves-with-ir-markers-for-improved-accurancy/1726


    Right, our Tools API is designed for long, thin objects like chopsticks, pencils, or drumsticks :D By opening up the raw data, the Image API lets you apply your own algorithms to identify and track objects. In the thread you linked, zalo points to a marker tracking script that he wrote that is able to track retroreflective stickers: https://community.leapmotion.com/t/retroreflective-marker-tracking-script-for-unity/1596 (I personally haven't seen any other community projects of this nature, so it's definitely worth checking out.)
    Head writer @ Leap Motion
    Up and Coming
    September 29, 2014
    Thanks Alex for taking the time to reply in detail.

    "alexcolgan" wrote:
    For the specific example you gave, our API makes it possible to attach IR markers to real-world objects and make them trackable in 3D space.

    Yes that could work for games where you always have a gun on your hand for instance. But i think that in VR we will instinctively want to explore the environments and manipulate the objects that we find on our way; for me the ideal input should make the transition between both use-cases (holding and not-holding something) seamlessly.

    "alexcolgan" wrote:
    At the moment, we haven't yet optimized our tracking algorithms for beta HMD-mode tracking. Mostly because they're evolving too fast for optimization to be worthwhile, as we're currently focused on improving other forms of tracking performance (e.g. more robust recognition of different hand poses). Ultimately, our goal (not just for latency, but for tracking overall) is that, if you're looking at the raw infrared imagery, the Leap Motion Controller's ability to recognize and follow your hands should be equal to (or even better than) your eyes.

    I'm making my assumptions based on the demo that i tried at Connect, on the main hallway. Latency was definitely noticeable, and it also had problems tracking the fingers when you could only see the side of your hand for example (which is understandable). I'm not sure what version of the software the demo was running on.. Anyway it's great to hear that you are improving it and that you are confident latency will be a non-issue.

    "alexcolgan" wrote:
    I definitely appreciate your skepticism. For what it's worth, we haven't experienced any issues over months of development and experimentation, and I haven't heard of any devs running into problems since we released the first VR build last month.

    I don't know what tests you have done but i can definitely break the Rift's positional tracking with my hands very easily, anyone with a DK2 can do the simple experiment of putting both hands between the HMD and the camera and move side to side. First thing that many people will try is to bring their hands in front and see how they react in VR, also certain interactions like drawing a bow or aiming a sniper rifle will require you to do so.


    Don't get me wrong i really appreciate the work that you guys are doing and wish you all the best. When everything finally works as intended and without hiccups i think it will be a good option to have, mainly for convenience and also for certain types of applications.

    But personally i still believe motion controllers tracked by 2 cameras (1 outside-in + 1 inside-out) with mechanically tracked fingers would be better for most use-cases, so i hope something close to this is what Oculus adopt as their main input method. Having said that, it doesn't have to be the only one!
    Up and Coming
    September 29, 2014
    Thanks PatimPatam, I really appreciate hearing your POV; it's times like these I wish I could jump into a time machine and look ten years in the future :D We're living in the middle of a Cambrian explosion and I can't wait to see how everything gets sorted out.

    Occlusion between the positional tracking camera and the DK2 shouldn't occur if you're angled a little bit to one side or another (which we currently recommend anyway, as reflections off monitors can degrade tracking slightly as opposed to facing open space). Because of the LEDs on the sides of the DK2, the camera can track position for quite a ways before you completely turn around. As far as I can tell, it only needs to see a few LEDs in order to work.

    Even beyond that, we generally don't recommend designing experiences that force users to bring their hands up to face level, since it gets tiring very quickly (the dreaded gorilla arm), so a camera mounted on top of your monitor should almost never be blocked.
    Head writer @ Leap Motion
    Up and Coming
    October 4, 2014
    I'm just going to leave these here:




    (This is an old video; I've refined the tracking since (though demoing it at Oculus Connect was painful because the room was covered in IR-interfering flurorescent lights)).


    And of course this:



    You can download and try it here; you'll want to use a shortcut to force dx11 though.

    Some of my coworkers discovered that if you draw on your hand with washable black marker, you can "key out" those areas of your hand, making for a really neat effect :)

    Some guy even liked it enough to record himself while commentating


    Additionally, there's little reason to be restricted to just keying out the hands from the AR feed; with some CV magic, it should be possible to key out objects like your keyboard and mouse as well. Visual SLAM aided by IMU data will be a lot easier with the dragonfly module... People haven't even begun to tap the possibilities of integrating CV techniques into actual human vision.