Forum Discussion
The VR Input Device Problem (and solution)
So I've been holding this idea in for a while, unsure how to actually make it a reality. Since I don't know how to proceed, I just wanted to throw it out there for everyone and see what happens. Maybe nobody thinks its a good idea but who knows.
First I just want to lay out the set of assumptions that lead me to the idea.
So the idea, my solution to the above set of assumptions is simply what I'm calling:
A Capacitive Keyboard for Visualization in Virtual Environments
Capacitive touch has been on keyboards for a while, but in all of those existing cases the capactive touch is used to send keyup/keydown messages to the PC. Instead what I propose is that we use a standard mechanical keyboard as we use today but add capacitive sensors to every key, not for sending key press signals, but for sending touch information. We could then, in the VR environment, display a 3D model of a keyboard probably someplace inconspicuous such as only when the user looks directly "down".
We then highlight the view of the virtual keyboard using the capacitive touch information being sent from the physical keyboard and this will allow us to create a feedback loop of tactile and visual input. The user will be able to align their hands on the home row easily as well as find hard to reach keys more easily. Looking back up would hide the keyboard restoring immersion. It would also allow you to view the keyboard un-occluded by your hands which may aid certain people in learning to touch type.
I believe the same idea could be translated to other hardware devices, such as mice, game pads and joysticks.
I believe that this is a fairly simple solution that doesn't require any additional research, would be inexpensive and would work with existing hardware and software systems, would have extremely low latency, extremely high accuracy and would not require much processing power to achieve. It is something any user could understand intuitively (assuming they are already familiar with the input device outside of VR), it would meet the requirements for a VR sit down experience and would be easy for software developers to adapt to.
The input device problem is one of the biggest hurdles for VR to overcome and I believe if we agreed that this was a good short term solution it would allow VR to move forward and continue to innovate on other input systems as longer term goals.
EDIT: added "VR is for much more than just games" to assumption list
First I just want to lay out the set of assumptions that lead me to the idea.
- Oculus is targeting a sit down experience in the short term
- VR is for much more than just games
- Using a keyboard while wearing an HMD is too difficult
- Using a camera to see real-world objects decreases immersion
- Using motion tracking to track hands or objects:
- has too much latency (for now)
- is not deterministic enough.
- has oclusion problems.
- has conflicts with room lighting or other ir devices.
- GamePads are great, but not for text input or other complex input scenarios.
- We want an input solution that doesn't require you to remove the HMD for entire sessions (hours)
So the idea, my solution to the above set of assumptions is simply what I'm calling:
A Capacitive Keyboard for Visualization in Virtual Environments
Capacitive touch has been on keyboards for a while, but in all of those existing cases the capactive touch is used to send keyup/keydown messages to the PC. Instead what I propose is that we use a standard mechanical keyboard as we use today but add capacitive sensors to every key, not for sending key press signals, but for sending touch information. We could then, in the VR environment, display a 3D model of a keyboard probably someplace inconspicuous such as only when the user looks directly "down".
We then highlight the view of the virtual keyboard using the capacitive touch information being sent from the physical keyboard and this will allow us to create a feedback loop of tactile and visual input. The user will be able to align their hands on the home row easily as well as find hard to reach keys more easily. Looking back up would hide the keyboard restoring immersion. It would also allow you to view the keyboard un-occluded by your hands which may aid certain people in learning to touch type.
I believe the same idea could be translated to other hardware devices, such as mice, game pads and joysticks.
I believe that this is a fairly simple solution that doesn't require any additional research, would be inexpensive and would work with existing hardware and software systems, would have extremely low latency, extremely high accuracy and would not require much processing power to achieve. It is something any user could understand intuitively (assuming they are already familiar with the input device outside of VR), it would meet the requirements for a VR sit down experience and would be easy for software developers to adapt to.
The input device problem is one of the biggest hurdles for VR to overcome and I believe if we agreed that this was a good short term solution it would allow VR to move forward and continue to innovate on other input systems as longer term goals.
EDIT: added "VR is for much more than just games" to assumption list
6 Replies
- I have a touch pad arm device that I am working on for personal use, but as I am already do one open source VR controller I suppose I could add it in too. Ill take some pictures of it so far....
~B - I think if you need really good text input, ie writing a book in VR, then there is no way around using a keyboard for optimal input control. I know some Leap Motion dev managed to get his keyboard into VR.
But for immersion, I'd say nooooooo to keyboard. It just reminds the player that they're still sitting in front of their computer, and controlling a virtual avatar in a virtual world with a keyboard is very difficult for people who aren't used to WASD-controls. So anything but keyboard if you're focusing on gameplay and immersion and reaching the masses.
I can't really think of a scenario with VR where you need really good text input though? I think a gamepad and a virtual keyboard (like Xbox has?) could work fairly well if you need text input.
Oculus is releasing their own controller at some point (either with CV1 or at a later time) which will use motion controller, so that will be interesting. "MrsVR" wrote:
I can't really think of a scenario with VR where you need really good text input though?
I'm operating under the assumption ('ll add it to the list) that VR is for much more than just games. So for an example scenario, imagine using a Facebook app. VR should be able to replace a classic 2D desktop for all work related functions as well. Such as coding, excel and yes writing a book. If for no other reason than the cost effectiveness of a single HMD vs. multiple physical monitors for effective screen real estate, VR will win.
Once you have committed the controls for a given app to muscle memory you effectively lose track of your hands and the device and you are able to focus on the game without losing immersion, this has been happening with gamepads and keyboards for a long time. There are all kinds of scenarios where you won't be having first person controls (Lucky's Tale for example), where needing to see your hands doesn't make sense (aliens without hands?), and how do people without hands use a hand tracking only system? Even without all of those limitations, using hand tracking just isn't deterministic enough yet. It's sort of like speech recognition software that has been around for decades but is never quite good enough. It's still waiting for some breakthrough to reach Iron Man "Jeeves" level quality and when that happens I'll be all over it. But for now we have to be realistic and realize that the leap, the kinect, the ps move, etc. are cool but gimmicky at best. Not practical for adopting as primary input systems at this time. Also, using a realworld camera in your game is about the biggest immersion breaker there is.
Furthermore another problem with hand tracking is its lack of haptic feedback. When you push out against something and in the game world you touch something but in the real world you touch nothing, its majorly immersion breaking. In cases like this an abstraction can be quite useful. Meaning "Press A to push", has been used in games for a long time and it works out pretty well. Gamepads will work really well in VR for a lot of scenarios (except complex text input), and keyboards will do the rest of the scenarios pretty well also. The only thing we need to do is to connect the feedback loop so the user can see the keyboard and it becomes practical for VR again, I believe."justinmchase" wrote:
"MrsVR" wrote:
<cut>
Furthermore another problem with hand tracking is its lack of haptic feedback. When you push out against something and in the game world you touch something but in the real world you touch nothing, its majorly immersion breaking. In cases like this an abstraction can be quite useful. Meaning "Press A to push", has been used in games for a long time and it works out pretty well. Gamepads will work really well in VR for a lot of scenarios (except complex text input), and keyboards will do the rest of the scenarios pretty well also. The only thing we need to do is to connect the feedback loop so the user can see the keyboard and it becomes practical for VR again, I believe.
<cut>
Good points. One comment I have is that haptic feedback can somewhat be "fooled" by a combination of audio and visual cues, e.g. lighting up the object you are touching and providing a "clicking" sound.
Cheers- now i realize the new thread i just posted is pointless i should of just searched "keyboard" and found this one..
lol
it can stay under the table then..(DK2)
i'll be back in 6 months again.
A Capacitive Keyboard for Visualization in Virtual Environments (exactly what is needed) - Here is the best solution !!! :
Imagine if you had a perfect 3d model of YOUR keyboard, and you calibrate the position manually with the virtual representation of the keyboard. So you see the virtual keyboard pinned in the virtual world, and you just reach out for it, and your hands land on the actual keyboard.
It feels so real !!!!
