Up and Coming

Leap Motion sets a course for VR

64 replies
12297 views

Open letter from Leap Motion CTO David Holz, posted this morning at https://community.leapmotion.com/t/leap-motion-sets-a-course-for-vr/1642:

Hi everyone,

I’d like to take a moment to talk about a series of developments we’ve been working on specifically for virtual reality. The first set involves our existing peripheral device and new things developers can do with it starting today. The second is a hint of some of our next-generation hardware and software efforts that we’re currently building from the ground up for this new and exciting space.

If virtual reality is to be anything like actual reality, we believe that fast, accurate, and robust hand tracking will be absolutely essential. We believe in the concept of other specialized controllers as well, but our hands themselves are the fundamental and universal human input device.

One of the most exciting things to us about virtual reality is that our technology can be more than just your hands -- it can be your eyes as well. We’ve recently hinted at this with the release of a new API which opens up raw infrared imagery24 straight from our sensors. When mounted directly onto a head-worn display, these images become stereoscopic windows into the world around you. What it sees, you see.

This expands the tracking space to be in any direction you’re facing. You can reach forward, turn around, look up and down, and the tracking follows you wherever you go. Because our device’s field of view exceeds that of existing VR displays, you’ll find it can start to track your hands even before you even see them.

To help people explore this paradigm with us, today we’re releasing a VR Developer Mount89, which allows you to easily and consistently attach and remove your Leap Motion Controller from a VR headset. We’re also releasing a software update for our beta SDK, which includes a massively improved ‘top-down tracking’ mode, as well as Unity and C++ examples. These show how to use both the image overlays and the tracking data from a head-mounted position, then give further examples of more sophisticated 3D interactions.

We think that this path fundamentally leads to a place where the digital and physical mediums of today blur together in deep and profound ways. Where bits and bytes become concepts and feelings and substances -- giving rise to digital interactions that are physically indistinguishable from real life.

To this end, I’d like to also give a hint of what we’re working on for the future. One prototype sensor that we’re beginning to show today (and will be giving out more information on in the future) is codenamed “Dragonfly.” It possesses greater-than-HD image resolution, color and infrared imagery, and a significantly larger field of view.

With next-generation “mega-sensors” like this, a Leap Motion device can literally become your eyes into the digital and physical realms -- allowing you to seamlessly mix and mash, fade and blend, between virtual environments and the sharpness of the real world.

Beyond the hardware, it’s important to stress that all forms of tracking are themselves software -- in our case, a deeply complex and rapidly evolving intelligence for tracking hands. We hope you will appreciate its growth as we continue to release updates every few weeks.

We’ve always been in awe of people’s deep and abiding passion for the dream of a digital and physical convergence. It’s this energy which drives us to work tirelessly until it is a reality, and we hope you continue to travel with us (or join us now) in this quest.

David

Show previous replies

M

MikeF

Inspiring

Have you noticed any loss of positional tracking with the mount in place?

I've got one on order but they've lost it in transit so its gonna be a while before i get to try it myself

S

Shannonb1

Up and Coming

"MikeF" wrote:
Have you noticed any loss of positional tracking with the mount in place?

I've got one on order but they've lost it in transit so its gonna be a while before i get to try it myself

THEY LOST MINE TOO!!! I have zero faith in the leap company....first they said it was on the way, then he was going to overnight it but retracted that because it was already out....then i found out it wasnt.....they suck....was supposed to get it wednesday now Im going to get it next monday....i could have used 3m tape and ordered a usb extension from amazon by now....another piece of crap company.

G

Gerald

Up and Coming

mine came through with no problem - lying here now 2 weeks still unopened, I just can't bring myself to glue anything to my DK2. waiting for top-down support to work from a stationary position (like my monitor) since I do not want to have to look at my hands.

check out my Mobile VR Jam 2015 title Guns N' Dragons

M

MikeF

Inspiring

"shannonb1" wrote:
"MikeF" wrote:
Have you noticed any loss of positional tracking with the mount in place?

I've got one on order but they've lost it in transit so its gonna be a while before i get to try it myself

THEY LOST MINE TOO!!! I have zero faith in the leap company....first they said it was on the way, then he was going to overnight it but retracted that because it was already out....then i found out it wasnt.....they suck....was supposed to get it wednesday now Im going to get it next monday....i could have used 3m tape and ordered a usb extension from amazon by now....another piece of crap company.

They sent mine out with DHL so i blame the choice of courier more than anything. a few grams of plastic shouldn't take more than a couple days to get anywhere. Their support has been great, but DHL blows.

I

Itsinthemind

Inspiring

I am expecting (and hoping) for the leap motion to become an integral part of future VR devices. The first thing I noted when friends are getting lost in my DK2 presentation, they try to find their hands. Being able to not just see but use your hands for interaction in VR of course is most natural and much be better than anything I can think of. I am keen to hear some kind of announcement from Oculus and then I am going to "Leap" at it and integrate it in my next project :)

http://www.lightandmagic.co.uk Fantasy becomes Reality

G

Gerald

Up and Coming

"Itsinthemind" wrote:
I am expecting (and hoping) for the leap motion to become an integral part of future VR devices. The first thing I noted when friends are getting lost in my DK2 presentation, they try to find their hands. Being able to not just see but use your hands for interaction in VR of course is most natural and much be better than anything I can think of. I am keen to hear some kind of announcement from Oculus and then I am going to "Leap" at it and integrate it in my next project :)

I do not see Oculus promoting a product of another company (especially not one of a company with a not so glamourous track record). So if Leap wants to get into VR, then they will have to achieve that on their own - and I think they are trying harder this time. But just as you - all developers are waiting.
I personally am waiting too - because why release anything now, maybe the big motion control wave will come 3 years from now with an entirely different product. There is nobody that buys the apps anyway. Have a look at the number of releases the Leap sees at all and you get an idea how attractive that market is for developers.

I am hoping Oculus will show a fresh input solution, and I would be very happy if it is a solution that I can port my so far extensive bundle of VR Leap experiences to. But as a developer I currently rather focus on giving the Gear a shot - even though from tech and experience point of view I find the Leap far more fascinating!

check out my Mobile VR Jam 2015 title Guns N' Dragons

M

mptp

Up and Coming

I got my Leap a couple of days ago and have been messing around with the API in Unity and seeing what the tracking can do.

I was surprised to find that all of the videos I've seen have really downplayed the limitations in the tracking capabilities of the Leap. Don't get me wrong, it's a fantastic little device for $99, but it's definitely nowhere near good enough for consumer VR. I'm still waiting on my VR mount, once it arrives I'm going to do a thorough video so that anybody on the fence in regards to ordering can get a good idea of what the Leap can do, but I'll post a few things I've noticed here:

1. Finger position is almost arbitrary. There are some finger gestures (like if you have your hands stretched out with no fingers touching each other, and move fingers up and down a little) that work perfectly. But just about anything else is either sketchy (finger joint angles are anywhere from 0 to 45 degrees off) or completely broken (putting your hand into a fist only works at very specific angles)

2. Any occlusion at all will cause a hand to completely spaz out before dropping out of tracking

3. Once a hand gets 'locked' into an incorrect position, Leap is quite stubborn about reevaluating the hand pose. The fastest and most reliable way of fixing this is to just remove your hand from Leap's view and put it back.

4. If you turn your hand in any orientation where Leap can't see the whole length of each finger, tracking goes very jittery and jumpy as it tries to figure out what's going on.

That all being said, if you aren't too concerned with accurate finger tracking, Leap is really fantastic. I'm currently creating an app that only uses Hand.PalmPosition, Hand.Direction, Hand.PalmNormal and Hand.GrabStrength, and the Leap is really awesome for that.
Also, I find that it only takes a few minutes for you to intuitively grasp what the Leap's limitations are and limit your movements such that the tracking behaves nicely. I suspect that's why so many of these videos don't show these tracking limitations - I very quickly got used to moving my hands in such a way that they were tracked nicely.

P.S. I just remembered, and can't be bothered slotting it in somewhere that will look nice - the Tool tracking in Leap is really underrated. It tracks a pen I had laying on my desk spectacularly - probably better than it tracks my hands! Something to consider.

Melbourne-based creative technologist. I flit between experimental AR/VR experiences, audiovisual electronics and full-stack web development. http://www.lachansleight.io

G

Gerald

Up and Coming

"mptp" wrote:

That all being said, if you aren't too concerned with accurate finger tracking, Leap is really fantastic. I'm currently creating an app that only uses Hand.PalmPosition, Hand.Direction, Hand.PalmNormal and Hand.GrabStrength, and the Leap is really awesome for that.
Also, I find that it only takes a few minutes for you to intuitively grasp what the Leap's limitations are and limit your movements such that the tracking behaves nicely. I suspect that's why so many of these videos don't show these tracking limitations - I very quickly got used to moving my hands in such a way that they were tracked nicely.

P.S. I just remembered, and can't be bothered slotting it in somewhere that will look nice - the Tool tracking in Leap is really underrated. It tracks a pen I had laying on my desk spectacularly - probably better than it tracks my hands! Something to consider.

:lol: - I always love it when devs come to that conclusion. I am preaching that to them for over a year now, but they are so stuborn with the hands that they completely ignore the awesome things you can do that actually work well.
I talked to their CEO a few weeks back and my last words were "The hands will be your doom!" ... because they look nice in videos, but in VR it is much more useful to have a super accurate 3D mousepointer imo. And that works well. If I were you then I would switch away from the palm tracking - that does work well, but the strain when using it longer is just too big.
Use single finger tracking, that works really well too and you can have your elbow resting on the desk for it.
Currently I am pressing them to optimize top-down tracking, so that you do not have to mount it on your Rift but can mount it for example on your monitor. That way you could use the desk as a touchscreen, a virtual keyboard, a ton of other things ...
They have the greatest tool for VR but are ignoring the possibilities ... because "Hands"! ;)

check out my Mobile VR Jam 2015 title Guns N' Dragons

A

alexcolganAuthor

Up and Coming

@mptp Although the software has made rapid advances since we launched the VR public beta less than 3 weeks ago, head-mounted tracking mode is definitely still bleeding-edge. Besides the technical challenge of optimizing for a totally different hand orientation, the Leap tends to work best when facing open space, as reflective backgrounds, like white walls, can make the hand hard to distinguish. (You can basically see how this affects the sensor's view by firing up the Image API and pointing it around.)

We're constantly working to improve on this issue (and the other ones you mentioned!) with new software updates, so stay tuned ;)

Head writer @ Leap Motion

M

mrgreen72

Inspiring

So on its way from San Diego to Canada, my Leap mount thought it'd stop by Germany... :lol: