Rift-Mounted Leap Motion Controller Demo (WIP 4: 5/9/13)

The issue is not with the Leap being mobile, but that it is fundamentally not a depth-sensor lime the Kinect. There is no point cloud generated, internally on on the computer (the mockups showing a visualisation of a point cloud are VERY disingenuous), the Leap picks up line disparity between it's two cameras (looking for linear highlights on cylindrical objects) and uses the length, ends, and relative orientations to give vectors and positions relative to it.
This gives a relatively simple system with a very fast response on cheap hardware and no need for custom illuminators (the Kinect's structured light projector and Intel's time-of-flight sensor are hardly COTS components). The drawback is that it has no clue as to what the objects it's tracking actually are. In the case of this demo, if you hold up your left hand palm-up, look away (i.e. keep your hand in front of you and turn your head, and thus the leap's volume of track, away), lower your left hand and raise your right hand palm-down, then look back at your right hand, the Leap has no way of identifying if this is you left or right hand. With a depth-sensing camera, you can follow the arm and guess which side of the body it attaches to, or look visually at the hand e.g. to identify fingernails. The Leap cannot do this with the current SDK, and if the developer comments that they do not intend to allow access to the raw camera data (likely because the Leap hardware is not designed to expose this) continue, it will not be possible without a hardware revision.
It works as long as it can keep both your hands in view at all times and keep your fingers splayed. Once you close your fingers, or rotate your hands out of view, the Leap cannot identify which hand is which, or that hand's orientation. This may turn out to be only a minor UI annoyance (FFVII block-hands?), or the lack of continuous and accurate tracking may cause too much weird dissociation for it to be usable outside a few specific cases. I'm certainly watching with bated breath to see if it's a viable setup.

In response to me asking whether the point cloud used data not currently exposed, David replied:
Yes, we actually reconstruct the entire hand in 3D. We take this reconstruction segment it into fingers and tools and fit planes to it etc, etc, and turn that into the API data you get today.

However, the raw format of this reconstruction is not actually a point cloud and is extremely unwieldy (it would be like giving out point cloud data in the 80s, no one would know how to use it, and it would give away proprietary details about the tech which we would prefer to hold back until patents publish).

As skeletal features are finally pushed into the API we will allow this to be directly viewed inside of the visualizer, and this will have similar visual appeal.

Looks awesome!

"cybereality" wrote:
Looks awesome!

Looks like it came right out of Johnny Mnemonic! Now to enhance it with gloves and force feedback!

"kojack" wrote:
Maybe you should reread your thread again before accusing us of stifling your creativity. Only one person (the very last post) said it wouldn't work because the leap must be stationary. Most of the posts supported the idea. Some had concerns over chaining of sensor dependencies, but never said it wasn't possible. In fact Sebbi, jordanp and I talked about attaching the leap to the rift and you thought it wouldn't work as well as having the leap attached to your chest instead.

Again please don't think I felt "my creativity" was being stifled. It was simply an idea I had that I quite honestly would have had no way of bringing to fruition. And I remember you... You came up with this hum-dinger when I mentioned the Leap being able to track my out stretched arms: "You have arms like a tyrannosaurus?" (in other words there was no way the Leap could track an outstretched arm of anything less then a stubby armed extinct dinosaur). That was you wasn't it??? Although it would seem in the video his arms don't appear to be "tyrannosaurus" like. Sorry to be so sensitive to your obviously positive feedback. :|

p.s. I know not EVERY comment was like this but I walked away from the thread with the feeling that what @zalo created wouldn't be possible. It had NOTHING to do with MY creativity. Just my hope for a cool way to interact in VR.

If you meant outstretched in front of you then it probably would have worked (just be at the edge of the leap range). I was thinking of outstretched to either side. I was mainly referring to your comment: "This might be an issue for someone with crazy long arms but I am around 5'10" and my arms rotate around a space quite close to 8 cubic feet". Since 8 cubic feet is only 2 feet x 2 feet x 2 feet and you'd need to measure from where the rift was placed (centre of chest or similar) my first thought was tyrannosaurus arms. :)
So I wasn't doubting the idea, just the amount of arm movement it could see. I should have been more clear on that, sorry.

Although actually the leap's vision is a squashed hemisphere, so it's range would be greater on two axes and shorter on the other. It's a bit hard to really understand what area it can cover without trying one out first hand (which I haven't done, I'm just going by forum specs).

"kojack" wrote:
If you meant outstretched in front of you then it probably would have worked (just be at the edge of the leap range). I was thinking of outstretched to either side. I was mainly referring to your comment: "This might be an issue for someone with crazy long arms but I am around 5'10" and my arms rotate around a space quite close to 8 cubic feet". Since 8 cubic feet is only 2 feet x 2 feet x 2 feet and you'd need to measure from where the rift was placed (centre of chest or similar) my first thought was tyrannosaurus arms. :)
So I wasn't doubting the idea, just the amount of arm movement it could see. I should have been more clear on that, sorry.

Although actually the leap's vision is a squashed hemisphere, so it's range would be greater on two axes and shorter on the other. It's a bit hard to really understand what area it can cover without trying one out first hand (which I haven't done, I'm just going by forum specs).

It would seem that the OP of this thread has made some impressive headway considering the concerns raised in pass threads. Hopefully when Jurassic Park becomes reality (as some would think we were as close to as realizing VR) Oculus will be able make a T-Rex compatible version so we can test the Leaps capability to see ones arm span! :lol:

Thé leap definitely has a narrow cone of vision. It is definitely not calibrated to match your rift FoV in it's current form, but since it already goes shaky and loses tracking when you hit the edge of your rift FoV, I feel like doing a true calibration would only neuter its capabilities further.

It also loses tracking just before you reach full outstretched arm length. It's probably due to the distance and to the change in angle of your hand relative to the camera once you get out that far.

Until leap improves their detection at steeper angles, this demo will probably remain an unviable proof of concept. For now, the hydra is the magic bullet... Unless someone wants to try a crazy head tracking scheme by using the positions of two hydra controllers from both the leap and hydra's perspective to do head tracking...

Sword-like Blunt Sticks!



Here I move around the cube a lot to show spatial awareness and I tap it on most of its sides. It's the most stable demo yet, it's finally physically responsive, and it's a great time waster!