VR/physical hybrid interfaces + Leap Motion hand tracking
Hello all,
For my CAD data visualization VR workspace project, I am faced with the need to use complex and precise interfaces like a 3D mouse (sometimes also called "space mouse"; it looks like this) inside a VR space. This gives much finer control than any motion controller available right now - unless maybe going 2-3 orders of magnitude higher in terms of cost (I still plan to use motion controls but in conjunction with this more precise control device).
The discussion below actually could apply just as well to the use of a regular keyboard or any other comparable physical interface device while in a VR environment.
The problem arises when you try to access all these buttons while inside VR. It's two-fold :
1. You need to know where the meatspace interface is located while seeing only VR space
2. You need to know where you own hand is located relative to that interface
1. is relatively easy to solve : you put a VR replica of the physical interface at the same location relative to the camera (HMD). In the application I'm looking at, the user is seated and does not make use of head positional tracking, so it's easy to achieve - just enter the right coordinates for the VR model and you will have a precise recreation of your input device.
2. is trickier. You of course have direct haptic feedback from the physical device you are touching, but you will still be fumbling around for the right buttons because you can't really visualize where your fingers are. You can have the buttons on the VR device light up upon presses on the physical device and get a sense of where your hand is, but it's a posteriori feedback - what I want to do is know it instinctively before pressing anything.
You don't want to be wearing sensors on your hand that would interfere with your usage of the device. You also don't want a 3D model of your hand covering your view of the device itself (it's also a problem in meatspace : you can't see the buttons that are under your hand).
My tentative solution is the following : set up a Leap Motion device (or similar) directly above the input device, pointed down at your hand, and use the finger tracking data to display (semi-transparent) representations of your fingertips on top of the VR device.
If that works, you can have some interesting effects. For example, if I hover a finger over a specific button, I could have a contextual menu pop up showing me what that particular button does before I even touch it.
Extending to non-physical interfaces :
Assuming I can get the above to work, I'd like to take this further and get the possibility to call up virtual buttons in 3D space around the VR/physical hybrid interfaces. Those will have no physical equivalent, and appear as "holograms" to the VR user. They can be pressed with the Motion-tracked fingertips and could be located anywhere in the Motion detection field, which should cover a reasonable amount of space around the default hand placement. Since these menus can be completely contextual, you'd probably only need a few buttons at any given time, allowing to make them pretty big and not to hard to hit consistently. Those will have no haptic feedback... unless you can implement some crude form of it through a "rumble pack" (think vibrating game controller) somewhere around the user's wrist or even through their elbow if it's resting on the chair arm.
As of today this is nothing but ideas, especially since I do not own a Leap Motion yet. That device and its capabilities are the biggest unknown to me at this moment. I'd love to hear from someone having a bit of experience with the Leap if what I am describing here seems feasible with that piece of technology, or if I'm daydreaming. I'd also of course love to know what you guys think about the idea in general, independently from hardware. Are there such implementations in place already somewhere ? Would you find them useful ? What would you want from such a UI ?
For my CAD data visualization VR workspace project, I am faced with the need to use complex and precise interfaces like a 3D mouse (sometimes also called "space mouse"; it looks like this) inside a VR space. This gives much finer control than any motion controller available right now - unless maybe going 2-3 orders of magnitude higher in terms of cost (I still plan to use motion controls but in conjunction with this more precise control device).
The discussion below actually could apply just as well to the use of a regular keyboard or any other comparable physical interface device while in a VR environment.
The problem arises when you try to access all these buttons while inside VR. It's two-fold :
1. You need to know where the meatspace interface is located while seeing only VR space
2. You need to know where you own hand is located relative to that interface
1. is relatively easy to solve : you put a VR replica of the physical interface at the same location relative to the camera (HMD). In the application I'm looking at, the user is seated and does not make use of head positional tracking, so it's easy to achieve - just enter the right coordinates for the VR model and you will have a precise recreation of your input device.
2. is trickier. You of course have direct haptic feedback from the physical device you are touching, but you will still be fumbling around for the right buttons because you can't really visualize where your fingers are. You can have the buttons on the VR device light up upon presses on the physical device and get a sense of where your hand is, but it's a posteriori feedback - what I want to do is know it instinctively before pressing anything.
You don't want to be wearing sensors on your hand that would interfere with your usage of the device. You also don't want a 3D model of your hand covering your view of the device itself (it's also a problem in meatspace : you can't see the buttons that are under your hand).
My tentative solution is the following : set up a Leap Motion device (or similar) directly above the input device, pointed down at your hand, and use the finger tracking data to display (semi-transparent) representations of your fingertips on top of the VR device.
If that works, you can have some interesting effects. For example, if I hover a finger over a specific button, I could have a contextual menu pop up showing me what that particular button does before I even touch it.
Extending to non-physical interfaces :
Assuming I can get the above to work, I'd like to take this further and get the possibility to call up virtual buttons in 3D space around the VR/physical hybrid interfaces. Those will have no physical equivalent, and appear as "holograms" to the VR user. They can be pressed with the Motion-tracked fingertips and could be located anywhere in the Motion detection field, which should cover a reasonable amount of space around the default hand placement. Since these menus can be completely contextual, you'd probably only need a few buttons at any given time, allowing to make them pretty big and not to hard to hit consistently. Those will have no haptic feedback... unless you can implement some crude form of it through a "rumble pack" (think vibrating game controller) somewhere around the user's wrist or even through their elbow if it's resting on the chair arm.
As of today this is nothing but ideas, especially since I do not own a Leap Motion yet. That device and its capabilities are the biggest unknown to me at this moment. I'd love to hear from someone having a bit of experience with the Leap if what I am describing here seems feasible with that piece of technology, or if I'm daydreaming. I'd also of course love to know what you guys think about the idea in general, independently from hardware. Are there such implementations in place already somewhere ? Would you find them useful ? What would you want from such a UI ?