Forum Discussion
Mechanical exoskeleton: a New hand motion capturing approach
Hey Guys, my name is Aler, me and my team(Dexta Robotics @ "www.dextarobotics.com") have been working on a new type of hand motion capturing approach since last September. Over the year we made a lot of progress on making a truly affordable, easy to use and precise hand motion capturing device, called Dexmo. And we are super excited to share them with you since we finally got all patent applications filed...
Obviously hands are important. In virtual reality, seeing your hands gives you better immersion, and it allows you to truly interact with the virtual world: this is just the way it should be, not keyboards or mouse. But what's keeping us from embracing the digital world where we could see our hands? We believe it's the price and cost of data gloves.
According to our observation, most commercialized data-golves uses IMU(i.e. ControlVR, Noitom neurons) or bend-sensors(I.e.5DT, cyber gloves). Bend-sensors are expensive, and usually to regenerate a good hand model, the data-glove needs to have at least 9 bend-sensors(usually 14). The pure hardware cost of a bend sensor solution glove is over $200, and they are selling it for more than 5 times of that price. IMU is slightly cheaper, but IMU modules produce less noises in raw data generally cost higher. And that's probably what's keeping the price high.
Anyway, we figured out what's written above one year ago and have been thinking if their is a another way. If we do it differently, if we don't start from IMU or bend sensors, can we make it cheaper? And we found the answer: Mechanical Exoskeleton!!! I can't insert image here so here is the link to our prototypes:
Early Sketch:
http://i.imgur.com/eDxJj9D.jpg
Dexmo Original:
http://i.imgur.com/F0fvUJJ.jpg
Dexmo Alpha:
http://i.imgur.com/eMDlp2H.jpg
Dexmo Beta:
http://i.imgur.com/o9rIC5K.jpg
Please don't mind about the exposed wires, and how "not beautiful" they are(we get this comment a lot, but honestly, I love how they look).in the consumer version they will be fixed...
Please allow me to quickly explain 1.how it works and 2.why it can be made cheaper than all other methods and 3.why exoskeleton shows a better potential for force control feedback.
1.When the users move their fingers, the rod link structure would follow the user’s finger movement mechanically, then the movement would be reflected onto the rotational sensor of Dexmo and will be sent wirelessly to a computer via the on-board Bluetooth module. Our SDK will take the raw data, run them through our special kinematics hand regeneration algorithms, and then a skeletal hand model will be shown.
2.Noted we transform the bending of the finger into a rotational movement. And rotational sensors are stable, accessible and cheap.. I mean really cheap, a super high quality rotational sensor cost less than 2 dollars. All mechanical linkage can be made with injection molding, and in bulk quantity the plastic parts can be made really inexpensively.
3."There is more!", mechanical data gloves, due to its exoskeletonic nature, are very suitable for adding force-feedback systems. Our team is currently working on a lightweight, compact force-feedback systems that can be added easily on to dexmo. We will show you guys the pictures and videos when we made some huge progress.
We will put the detailed videos up later, now we only have one demo combo on the website showing some of our early demos. We are very excited about this technology, we spent a lot of time on it, and I hope you guys can like this as well!
Ok I need some sleep, it's 5 am in my current timezone...See you guys later, if you have any questions, feel free to ask, I will answer all of them when I wake up. If you can help spread the news for us, thousand of thanks, we would love to share this with more people!
Obviously hands are important. In virtual reality, seeing your hands gives you better immersion, and it allows you to truly interact with the virtual world: this is just the way it should be, not keyboards or mouse. But what's keeping us from embracing the digital world where we could see our hands? We believe it's the price and cost of data gloves.
According to our observation, most commercialized data-golves uses IMU(i.e. ControlVR, Noitom neurons) or bend-sensors(I.e.5DT, cyber gloves). Bend-sensors are expensive, and usually to regenerate a good hand model, the data-glove needs to have at least 9 bend-sensors(usually 14). The pure hardware cost of a bend sensor solution glove is over $200, and they are selling it for more than 5 times of that price. IMU is slightly cheaper, but IMU modules produce less noises in raw data generally cost higher. And that's probably what's keeping the price high.
Anyway, we figured out what's written above one year ago and have been thinking if their is a another way. If we do it differently, if we don't start from IMU or bend sensors, can we make it cheaper? And we found the answer: Mechanical Exoskeleton!!! I can't insert image here so here is the link to our prototypes:
Early Sketch:
http://i.imgur.com/eDxJj9D.jpg
Dexmo Original:
http://i.imgur.com/F0fvUJJ.jpg
Dexmo Alpha:
http://i.imgur.com/eMDlp2H.jpg
Dexmo Beta:
http://i.imgur.com/o9rIC5K.jpg
Please don't mind about the exposed wires, and how "not beautiful" they are(we get this comment a lot, but honestly, I love how they look).in the consumer version they will be fixed...
Please allow me to quickly explain 1.how it works and 2.why it can be made cheaper than all other methods and 3.why exoskeleton shows a better potential for force control feedback.
1.When the users move their fingers, the rod link structure would follow the user’s finger movement mechanically, then the movement would be reflected onto the rotational sensor of Dexmo and will be sent wirelessly to a computer via the on-board Bluetooth module. Our SDK will take the raw data, run them through our special kinematics hand regeneration algorithms, and then a skeletal hand model will be shown.
2.Noted we transform the bending of the finger into a rotational movement. And rotational sensors are stable, accessible and cheap.. I mean really cheap, a super high quality rotational sensor cost less than 2 dollars. All mechanical linkage can be made with injection molding, and in bulk quantity the plastic parts can be made really inexpensively.
3."There is more!", mechanical data gloves, due to its exoskeletonic nature, are very suitable for adding force-feedback systems. Our team is currently working on a lightweight, compact force-feedback systems that can be added easily on to dexmo. We will show you guys the pictures and videos when we made some huge progress.
We will put the detailed videos up later, now we only have one demo combo on the website showing some of our early demos. We are very excited about this technology, we spent a lot of time on it, and I hope you guys can like this as well!
Ok I need some sleep, it's 5 am in my current timezone...See you guys later, if you have any questions, feel free to ask, I will answer all of them when I wake up. If you can help spread the news for us, thousand of thanks, we would love to share this with more people!
19 Replies
- I think it looks great just the way it is! ....however I know why people complain, and besides the fact that some may not like that look, anything that dangles is something that can be caught on something else, or pulled off accidentally. I think they look awesome though! I can't wait to try them out. Our company here is just getting off the ground. We are currently designing a CNC mill just so I can mill my own frame for our 3D printer we will need, and then off to getting a kick-starter going hopefully get some start-up funds. I am hoping to partner with other hardware programmers and engineers, or anyone that wants to work on making VR possible for everyone. Feel free to send me a PM if you are interested in hearing more at some point. Great job, and keep up the awesome work!!
- Love your mechanical solution and I love the look too.
I can see how this works for the hand alone but what about the arms and wrists? Should this product be matched with other companies devices or will you be solving that with your own products? - Wow I really like your Dexmo Original design it looks like a cyberpunk glove it's really cool but I don't think it is really practical, the device is really bulky and pops out from your hand too much specially the beta one. Is it really comfortable when you cross your hands or do they get stuck when you do so ?
Anyway the sensor idea is really nice, we actually dont really need IMUS for every single finger joint. Have you tried maybe using small pulleys and wires through the fingers to get a sleeker and smaller desing? "lmaceleighton" wrote:
I think it looks great just the way it is! ....however I know why people complain, and besides the fact that some may not like that look, anything that dangles is something that can be caught on something else, or pulled off accidentally. I think they look awesome though! I can't wait to try them out. Our company here is just getting off the ground. We are currently designing a CNC mill just so I can mill my own frame for our 3D printer we will need, and then off to getting a kick-starter going hopefully get some start-up funds. I am hoping to partner with other hardware programmers and engineers, or anyone that wants to work on making VR possible for everyone. Feel free to send me a PM if you are interested in hearing more at some point. Great job, and keep up the awesome work!!
Thanks! The beta version does look kind of bulky, but it was a test version, the consumer version will be much smaller in size and more duable. Good luck with your kickstarter! We plan to launch ours on late September or early October:)"Colm" wrote:
Love your mechanical solution and I love the look too.
I can see how this works for the hand alone but what about the arms and wrists? Should this product be matched with other companies devices or will you be solving that with your own products?
Thanks! I love the looks too haha, about the space position tracking, excellent question!
We have built a device called the Dexmo Arm to get the absolute position in space.
It looks like this:http://i.imgur.com/MVc47Hx.jpg
basically you perform a forward kinematics calculation then you can get the absolute position in 3D along with the posture of Dexmo. But this was designed for robotics control(I will attach a youtube video later). Due to the limited work space the Dexmo arm might be more suitable for dexterous robotics control.
So then we shifted to IMU and optical solutions, there are two companies we are working in conjunction with, called X-immerse(a company that works on optical finger tip locating) and Noitom.(They recently launched a kickstarter, the Noitom Neutrons!) We are working on two type of position finding method.
1. X-immerse solution: you wear two cameras on Oculus Rift, and put three infrared LED on the tip of the glove with Dexmo (also on on the back of your hand), then their algorithm would calculate and adjust the coordinate system with the Oculus to find the absolute position of Dexmo.
2.Noitom IMU solution: similar to controlVR, we use 5 Noitom neutrons to regenerate the model of both arms. Thanks for asking!"NoxWings" wrote:
Wow I really like your Dexmo Original design it looks like a cyberpunk glove it's really cool but I don't think it is really practical, the device is really bulky and pops out from your hand too much specially the beta one. Is it really comfortable when you cross your hands or do they get stuck when you do so ?
Anyway the sensor idea is really nice, we actually dont really need IMUS for every single finger joint. Have you tried maybe using small pulleys and wires through the fingers to get a sleeker and smaller desing?
Yep the design you suggested had been tested four month ago by us and our conclusion is that pulleys and wires are not the most reliable hand motion capturing method. They can be made slightly smaller but really fragile, and it is supper painful to get the linear variable resistor back to its zero position, you know with all the springs and such, it's not really worth it. It's true this design might look a bit large at first sight, but hey, you won't be watching them with your Oculus on ;)
In terms of comfortableness, we let some friends tried out Dexmo alpha and beta and the feedback was pretty positive. We will be attending the Oculus Connect, if you are around LA by then we totally welcome you to try it out yourself, the dexmo we will take there will looks more like a consumer version:D- Holy shit.
I just saw this on reddit and immediately came over here to post it because I was so excited. Looks like I was too slow. :D
You know, before I learned about IMU data gloves, I was interested in the exact same thing! It occurred to me that simple rotational sensors are damn cheap, and don't drift ever, so with some clever mechanical engineering, one could turn all finger motion into many single-axis rotations detected by dirt-cheap rotational sensors and have the perfect data glove!
I then realised that I have absolutely nowhere near the required level of expertise, so I gave up on the problem.
But you've gone and done it, and it looks super awesome! I have a couple of questions:
1. Do you have a ballpark estimate on how much a pair of these gloves are going to cost? Obviously not a final consumer price, but are we talking under $100, under $200, under $300?
2. Are you planning on making them wireless? (Please please make them wireless)
3. What is the latency currently like from mechanical movement of the sensors to reconstructed 3D hand pose?
4. Have you considered combining these gloves with something like a STEM pack to obtain the absolute position and rotation of the wrist, and use Dexmo for the finger motion? I'm fairly confident that in the future many many people will have a STEM system, or something resembling it, and this isn't prone to the issues involved with optical tracking.
5. Do you perhaps have any videos of Dexmo use that displays a reconstructed hand model on a screen? The telepresence video of you guys controlling the robotic hand is cool and all, but I'm personally more interested in seeing the software application than the hardware application. ;)
But yeah, this rocks my world. Thanks so much for making this!! - It will be a non-mechanical approach that wins the VR hand tracking race.
It will be mechanics that win the haptic hand interface race.
I believe an efficient approach for tracking hand movement will be found in the field of optics, most likely with easy to put on, and take off optical bracelets.
There are constraints in the way a hand may move. Further more, for each unique hand state, we might/will find that there are "landmark" positions of that handstate that only exist at that point in time (Maybe the specific joint position of your index finger, or knowing the way in which your ring and pinky finger are co-related in movement). Tracking/searching such states would be the fastest, low-cpu utilization approach to approximating hand position in a minimally invasive fashion.
And remember, the name of the game is minimal setup, and presence. I don't want to put on gloves every time I game. - Alpha looks much much better...
Good luck ! "nuB" wrote:
It will be a non-mechanical approach that wins the VR hand tracking race.
It will be mechanics that win the haptic hand interface race.
I believe an efficient approach for tracking hand movement will be found in the field of optics, most likely with easy to put on, and take off optical bracelets.
There are constraints in the way a hand may move. Further more, for each unique hand state, we might/will find that there are "landmark" positions of that handstate that only exist at that point in time (Maybe the specific joint position of your index finger, or knowing the way in which your ring and pinky finger are co-related in movement). Tracking/searching such states would be the fastest, low-cpu utilization approach to approximating hand position in a minimally invasive fashion.
And remember, the name of the game is minimal setup, and presence. I don't want to put on gloves every time I game.
I agree that at the end of the day, mechanical hand tracking isn't going to be the final solution. Ultimately it's too bulky and invasive. But optical finger-tracking will never be good enough without the use of multiple cameras, simply because of how much occlusion you get in basic hand movements. Mounting something onto your wrist (which is just as invasive as putting a glove on IMO - the whole attraction of optical is there's no requirement to 'put on' anything) is all well and good until you bend your wrist all the way forward/back and thus have your forearm blocking the camera's view of the hand.
I'm about 80% sure that without some new breakthrough technology, IMU hand tracking is where it's going to be at. Sure it's expensive now, with a set of Neuron gloves costing vaguely $200, but this cost will only go down - once it's under $100 then it won't be a huge investment. There's no occlusion issues, and latency is definitely acceptable, and will only get better.
The problem is, that's going to be years away, maybe 3, maybe 5, maybe 8...who knows? The point is, it's going to be too far away for good hand-tracking to be approachable to the everyday VR enthusiast. What we need right now is a good hand-tracking solution that costs less than that, which is why I'm so excited about Dexmo, and it's also why I'm so interested in knowing how much this thing is going to cost. Ultimately, it all comes down to price right now with hand-tracking solutions (when deciding between mechanical or IMU based - optical simply isn't good enough right now)
P.S. Another question for Davideus - are you planning on (perhaps in future versions) enabling the distinct detection of the rotation of the first and second joints? For example, the ability to make a 'quote quote' sign by only bending the second finger joints.
