Skip to main content
alexcolgan
Up and Coming
August 28, 2014

Leap Motion sets a course for VR

Open letter from Leap Motion CTO David Holz, posted this morning at https://community.leapmotion.com/t/leap-motion-sets-a-course-for-vr/1642:




Hi everyone,

I’d like to take a moment to talk about a series of developments we’ve been working on specifically for virtual reality. The first set involves our existing peripheral device and new things developers can do with it starting today. The second is a hint of some of our next-generation hardware and software efforts that we’re currently building from the ground up for this new and exciting space.

If virtual reality is to be anything like actual reality, we believe that fast, accurate, and robust hand tracking will be absolutely essential. We believe in the concept of other specialized controllers as well, but our hands themselves are the fundamental and universal human input device.

One of the most exciting things to us about virtual reality is that our technology can be more than just your hands -- it can be your eyes as well. We’ve recently hinted at this with the release of a new API which opens up raw infrared imagery24 straight from our sensors. When mounted directly onto a head-worn display, these images become stereoscopic windows into the world around you. What it sees, you see.

This expands the tracking space to be in any direction you’re facing. You can reach forward, turn around, look up and down, and the tracking follows you wherever you go. Because our device’s field of view exceeds that of existing VR displays, you’ll find it can start to track your hands even before you even see them.

To help people explore this paradigm with us, today we’re releasing a VR Developer Mount89, which allows you to easily and consistently attach and remove your Leap Motion Controller from a VR headset. We’re also releasing a software update for our beta SDK, which includes a massively improved ‘top-down tracking’ mode, as well as Unity and C++ examples. These show how to use both the image overlays and the tracking data from a head-mounted position, then give further examples of more sophisticated 3D interactions.

We think that this path fundamentally leads to a place where the digital and physical mediums of today blur together in deep and profound ways. Where bits and bytes become concepts and feelings and substances -- giving rise to digital interactions that are physically indistinguishable from real life.

To this end, I’d like to also give a hint of what we’re working on for the future. One prototype sensor that we’re beginning to show today (and will be giving out more information on in the future) is codenamed “Dragonfly.” It possesses greater-than-HD image resolution, color and infrared imagery, and a significantly larger field of view.

With next-generation “mega-sensors” like this, a Leap Motion device can literally become your eyes into the digital and physical realms -- allowing you to seamlessly mix and mash, fade and blend, between virtual environments and the sharpness of the real world.

Beyond the hardware, it’s important to stress that all forms of tracking are themselves software -- in our case, a deeply complex and rapidly evolving intelligence for tracking hands. We hope you will appreciate its growth as we continue to release updates every few weeks.

We’ve always been in awe of people’s deep and abiding passion for the dream of a digital and physical convergence. It’s this energy which drives us to work tirelessly until it is a reality, and we hope you continue to travel with us (or join us now) in this quest.

David

    64 replies

    Show previous replies
    zalo
    Up and Coming
    August 30, 2014



    I'm currently getting the example approved on the Leap Gallery so you all can play around with its source ;)

    In the mean-time, here's the demo executable: https://app.box.com/s/8pvtwkib55ylcdqqdehp
    fabsterpal
    Up and Coming
    August 31, 2014
    It'd be cool if the oculus rift detected the leap motion's IR leds and tracked them.

    That said for all my experiments I've used the door that covers the leads as an extension mount. Just scan it in 3D and print a to-scale (hard to get right) copy, then you can start adding brackets to it. It's also by the extension USB port, so less lead requirement (if you can get it to work). It's not blocking (m)any IR leds this way.
    3dvrmonster
    Up and Coming
    August 31, 2014
    "zalo" wrote:



    I'm currently getting the example approved on the Leap Gallery so you all can play around with its source ;)

    In the mean-time, here's the demo executable: https://app.box.com/s/8pvtwkib55ylcdqqdehp


    Very cool and thanks for sharing. I will check it out once my leap kit arrives.
    Intel Core i7-3820 OC@ 4.7 Ghz (Liquid-cooled) on ASUS Sabertooth X79 mobo 16 GB Quad Channel DDR 3 1600 Mhz with 250 gb SSD GTX 980 working with DK2 to make something great!
    alexcolganAuthor
    Up and Coming
    September 1, 2014
    "selkess" wrote:
    will hands work out of the box in rift demos or not ? or does it only work in leap motion own demos / games ?

    The hands only appear when developers add them to their apps, as they're part of our free Unity assets: https://www.assetstore.unity3d.com/en/#!/content/17770

    Game engines like Unity don't have any built-in assumptions about how hands are supposed to work in a scene, so we're working on these (and other!) resources to make it easier to drop motion controls into existing projects.
    Head writer @ Leap Motion
    mhenriquecd
    Participant
    September 2, 2014
    This can be easily replaced by a camera in own Oculus rift (the same site where it positioned the Lead Motion)
    Oculus VR could implement this,, help other camera (the front) to recognize both hands as a possible motion control!
    nuB
    Participant
    September 3, 2014
    The only application I see for a mounted Leap controller on an HMD, is for tracking mouth/lip movement for translation to virtual avatars, vastly increasing multiplayer VOIP realism.

    Using this to replace the IR camera is an overly expensive solution to a problem that doesn't really exist, and most likely will be fiscally addressed for CV1, or using a a 3D free space tracking device, which will become the norm eventually.
    Shannonb1
    Up and Coming
    September 5, 2014
    "nuB" wrote:
    The only application I see for a mounted Leap controller on an HMD, is for tracking mouth/lip movement for translation to virtual avatars, vastly increasing multiplayer VOIP realism.

    Using this to replace the IR camera is an overly expensive solution to a problem that doesn't really exist, and most likely will be fiscally addressed for CV1, or using a a 3D free space tracking device, which will become the norm eventually.



    This comment makes no sense to me? Your lips? No this is perfect for interacting with in game huds, switches, controls, push a door open, use a ladder. Your imagination is limited if the "only" application you can see is mouth/lip.

    youre also better off using a voice to mocap type solution for mouth movement in multiplayer. Much nicer on processing power and more accurate in my opinion.

    I dont think 80$ is an unreasonable price for what can be seen as a joystick peripheral. I paid 500$ for a wheel why wouldnt i pay 80 to see my arms and hands and interface naturally with the game.
    nuB
    Participant
    September 5, 2014
    "shannonb1" wrote:
    "nuB" wrote:
    The only application I see for a mounted Leap controller on an HMD, is for tracking mouth/lip movement for translation to virtual avatars, vastly increasing multiplayer VOIP realism.

    Using this to replace the IR camera is an overly expensive solution to a problem that doesn't really exist, and most likely will be fiscally addressed for CV1, or using a a 3D free space tracking device, which will become the norm eventually.



    This comment makes no sense to me? Your lips? No this is perfect for interacting with in game huds, switches, controls, push a door open, use a ladder. Your imagination is limited if the "only" application you can see is mouth/lip.

    youre also better off using a voice to mocap type solution for mouth movement in multiplayer. Much nicer on processing power and more accurate in my opinion.

    I dont think 80$ is an unreasonable price for what can be seen as a joystick peripheral. I paid 500$ for a wheel why wouldnt i pay 80 to see my arms and hands and interface naturally with the game.


    I don't think you've considered the constraints in Leap hand tracking.
      The view of hands must be unobstructed, or mostly so
      The view space in which your hands must be in to be tracked is not large enough for free hand movement around your body
      Overall on average, VR environments will have users holding peripherals, most of which will track in 3D space themselves
    Environments that satisfy the above conditions will not make up the bulk of the eventual VR market imo.

    I also don't think you understand the benefits I am alluding to when using the leap for 3D capture of mouth movement.
    Motion capture is needlessly bandwidth heavy. It would be simpler to condense mouth and lip movement into a model, and transmit that condensed data (think of it like data compression). Which is the main point of the Leap controller, interpreting and processing 3D data in real time. A version of this is already demonstrated with the hands, as Leap has decided that breaking down the 3D video into skeleton models of the hands is the best approach for handling hand movement data. I agree with them, and feel the same would apply to your mouth and lips.

    Lastly, as for replacing the IR camera, and assuming the person doesn't have any sort of 3D free space tracking peripheral, I ask how would the leap controller be better? The IR camera is detached from the user, providing a solid, unmoving anchor point to calculate an HMD's absolute position from (no chance of drift). Using the Leap on the HMD for the same purpose requires that the interpreted 3D model of the environment is calculated perfectly to simply match the effectiveness of the IR camera. Of course you cannot turn around fully with camera tracking, and that is indeed a benefit the Leap controller could provide (or an IR LED attached to the back of your head), but it is such a small one, which is only plausible when the user doesn't have something like a Stem pack, or an alternative product (which is going to be common place in consumer VR), to replace the IR camera for head/HMD tracking.

    This post should further illustrate the boundaries of my imagination regarding this thread/topic. If you are able to directly refute the logic of anything I am trying to express, feel free, it will be to my benefit.
    Shannonb1
    Up and Coming
    September 6, 2014
    Will do as soon as I'm off phone and on PC.
    Anonymous
    September 10, 2014
    I got the Leap Motion Rift mount Monday and set it up. Today I tried the Plasma Ball VR example. Very impressed.
    https://developer.leapmotion.com/gallery/plasma-ball-vr
    Show more replies
    Terms & ConditionsAccessibility statement