Forum Discussion
Face Tracking - Possibilities and Uses
Some of you may be aware of this kind of tech, but here are some examples anyway! :
http://www.youtube.com/watch?v=CvievLyt ... OItKHmairQ - Standard Webcam
http://www.youtube.com/watch?v=jE-gftsaXb8 -Kinect for Windows
I was going through some possibilities and problems surrounding face tracking in VR with Head Mounted Displays a while ago and thought I could share them for discussion.
As I don't have my Dev Kit yet, I have not set up any demos or run many tests. I have however experimented with Kinect and Webcam face tracking.
Before I run through some exciting possibilities, I think it's best to go through some problems developers may encounter if they wish to create face tracking applications for VR games or simulations.
Much of this may be on speculation as I'm yet to combined the Rift with face tracking tech. Please feel free to try it yourself, I will only get a little bit jealous ;).
Please feel free to admire the beautiful names I've designated them :D :
The Half Face Problem
It should come at no surprise that the rift will cover a considerable amount of your face. In fact, the only useful area to track that is left uncovered is your mouth. This means that unless VR HMD's suddenly and magically can be reduced to the size of your eyes alone leaving much of your face uncovered, you will only be able to track mouth movements with an external webcam or Kinect. Now this has its benefits of course, but let's save that all for later. What if you want to track eye/eyebrow movements?
That's the problem I'm having. The only solution I can possibly come up with is some kind of internal tracking device inside the HMD itself. The major issue with that would be space. Last time I checked, Optical Devices required a decent enough amount of room (Although smartphone cameras are getting smaller and smaller) and enough light to preform correctly. Although I can't help but wondering that given a good enough sensor (compact and powerful), you could accurately track eye/eyebrow movements well enough to mimic them with a 3d model/ Character.
Again, this is all based on speculation. I don't know for myself how much space is inside the Rift itself, or if it's possible to insert a sensor in that way.
The OMFG STOP MOVING Problem
The Rift has sensational head tracking capabilities that are not even matched in consumer HMD's. This provides an accurate and low latency control scheme while adding to the immersion. It's very VERY awesome. But for face tracking, I can imagine it being a nightmare. (Not that face tracking is more important, I'm just proving a point :D)
Consider you are tracking mouth movements with an external device (webcam or kinect). A player turns their head 90 degrees or so in any direction. How do you track their facial movements at that given moment? The facial tracking technology I've seen so far requires the user to be facing the optical device almost directly for constant tracking (obviously).
I'm having trouble designing a solution for this particular problem that doesn't require additional hardware such as repositioning webcams or multiple optical devices.
I will continue to outline the issues surrounding face tracking when using HMD's. I'm also starting a simple wordpress blog to log any thoughts or experiences I have with the Rift. This blog is not designed to attract traffic or generate revenue. I will not be hosting advertisements and I'm very unlikely to accept donations. It is no where near completion and I will not start posting for a few days, but here it is anyway:
http://usingtherift.wordpress.com/
Right. Enough of the problems and concerns. What are the applications for face tracking in VR?
This is the exciting part!
First off, let's talk about manipulating your character's expressions using your own in game. Imagine for example playing online with a friend. Imagine helping them out and actually being able to smile at them. Imagine pushing them off the level and laughing at them. Imagine pulling the strangest faces available just to creep them out. Social aspects are perhaps the most obvious use of this technology, but certainly an interesting one.
Let's not forget the possibilities of face tracking in Single Player. On the SXSW Oculus Pannel, I believe Chris Roberts mentioned that his team wanted to implement an eye-contact based conversation system for NPC's in the upcoming title Star Citizen. That concept alone is very impressive. But combined with advanced expression capabilities you could interact with NPC's like never before. How about having a serious discussion on politics with the galaxy's finest, smiling and pulling faces all the way through, only to be told off and removed from the meeting? I don't know about you, but that comical example alone sounds pretty deep and impressive to me.
Face tracking does not just have to be used for animating expression, however. This technology could be used as another Input/Control scheme. Imagine a game that puts you in the role of a fierce dragon. You are soaring high across a battlefield and target a legion of enemies. You then open your mouth as if to yawn, (but with violent intent :D), and fire explodes from the dragons mouth in game.
Or how about a game such as The Elder Scrolls V: Skyrim? An update to the Xbox 360 Version allowed users to use voice commands through a Kinect device that could use items or activate Dragon Shouts. I've even tried this but was not able to enjoy the experience without sounding completely crazy to those nearby. With mouth tracking alone, you could set up a simple input scheme that would allow users to activate shouts/abilities by mouthing them.
I will conclude this subject either here in the reply section or on my Using the Rift blog as mentioned above. But for now I should probably stop typing as I've had a long day and a lot of this is probably incorrect English as it stands. If I continued I would appear to have no vocabulary whatsoever :D.
I've not checked my post for mistakes thoroughly, so apologies in advance -_-.
Thanks for actually coping with my inability to form a post and I hope I've inspired some interesting experiments. Or at least got people considering face tracking in VR Videogames and Experiences.
http://www.youtube.com/watch?v=CvievLyt ... OItKHmairQ - Standard Webcam
http://www.youtube.com/watch?v=jE-gftsaXb8 -Kinect for Windows
I was going through some possibilities and problems surrounding face tracking in VR with Head Mounted Displays a while ago and thought I could share them for discussion.
As I don't have my Dev Kit yet, I have not set up any demos or run many tests. I have however experimented with Kinect and Webcam face tracking.
Before I run through some exciting possibilities, I think it's best to go through some problems developers may encounter if they wish to create face tracking applications for VR games or simulations.
Much of this may be on speculation as I'm yet to combined the Rift with face tracking tech. Please feel free to try it yourself, I will only get a little bit jealous ;).
Please feel free to admire the beautiful names I've designated them :D :
The Half Face Problem
It should come at no surprise that the rift will cover a considerable amount of your face. In fact, the only useful area to track that is left uncovered is your mouth. This means that unless VR HMD's suddenly and magically can be reduced to the size of your eyes alone leaving much of your face uncovered, you will only be able to track mouth movements with an external webcam or Kinect. Now this has its benefits of course, but let's save that all for later. What if you want to track eye/eyebrow movements?
That's the problem I'm having. The only solution I can possibly come up with is some kind of internal tracking device inside the HMD itself. The major issue with that would be space. Last time I checked, Optical Devices required a decent enough amount of room (Although smartphone cameras are getting smaller and smaller) and enough light to preform correctly. Although I can't help but wondering that given a good enough sensor (compact and powerful), you could accurately track eye/eyebrow movements well enough to mimic them with a 3d model/ Character.
Again, this is all based on speculation. I don't know for myself how much space is inside the Rift itself, or if it's possible to insert a sensor in that way.
The OMFG STOP MOVING Problem
The Rift has sensational head tracking capabilities that are not even matched in consumer HMD's. This provides an accurate and low latency control scheme while adding to the immersion. It's very VERY awesome. But for face tracking, I can imagine it being a nightmare. (Not that face tracking is more important, I'm just proving a point :D)
Consider you are tracking mouth movements with an external device (webcam or kinect). A player turns their head 90 degrees or so in any direction. How do you track their facial movements at that given moment? The facial tracking technology I've seen so far requires the user to be facing the optical device almost directly for constant tracking (obviously).
I'm having trouble designing a solution for this particular problem that doesn't require additional hardware such as repositioning webcams or multiple optical devices.
I will continue to outline the issues surrounding face tracking when using HMD's. I'm also starting a simple wordpress blog to log any thoughts or experiences I have with the Rift. This blog is not designed to attract traffic or generate revenue. I will not be hosting advertisements and I'm very unlikely to accept donations. It is no where near completion and I will not start posting for a few days, but here it is anyway:
http://usingtherift.wordpress.com/
Right. Enough of the problems and concerns. What are the applications for face tracking in VR?
This is the exciting part!
First off, let's talk about manipulating your character's expressions using your own in game. Imagine for example playing online with a friend. Imagine helping them out and actually being able to smile at them. Imagine pushing them off the level and laughing at them. Imagine pulling the strangest faces available just to creep them out. Social aspects are perhaps the most obvious use of this technology, but certainly an interesting one.
Let's not forget the possibilities of face tracking in Single Player. On the SXSW Oculus Pannel, I believe Chris Roberts mentioned that his team wanted to implement an eye-contact based conversation system for NPC's in the upcoming title Star Citizen. That concept alone is very impressive. But combined with advanced expression capabilities you could interact with NPC's like never before. How about having a serious discussion on politics with the galaxy's finest, smiling and pulling faces all the way through, only to be told off and removed from the meeting? I don't know about you, but that comical example alone sounds pretty deep and impressive to me.
Face tracking does not just have to be used for animating expression, however. This technology could be used as another Input/Control scheme. Imagine a game that puts you in the role of a fierce dragon. You are soaring high across a battlefield and target a legion of enemies. You then open your mouth as if to yawn, (but with violent intent :D), and fire explodes from the dragons mouth in game.
Or how about a game such as The Elder Scrolls V: Skyrim? An update to the Xbox 360 Version allowed users to use voice commands through a Kinect device that could use items or activate Dragon Shouts. I've even tried this but was not able to enjoy the experience without sounding completely crazy to those nearby. With mouth tracking alone, you could set up a simple input scheme that would allow users to activate shouts/abilities by mouthing them.
I will conclude this subject either here in the reply section or on my Using the Rift blog as mentioned above. But for now I should probably stop typing as I've had a long day and a lot of this is probably incorrect English as it stands. If I continued I would appear to have no vocabulary whatsoever :D.
I've not checked my post for mistakes thoroughly, so apologies in advance -_-.
Thanks for actually coping with my inability to form a post and I hope I've inspired some interesting experiments. Or at least got people considering face tracking in VR Videogames and Experiences.
4 Replies
- You could have a three-camera solution: two eye-tracking cameras (nIR, internal to the HMD) to track gaze direction, and possibly eyebrow position, and one final camera mounted below the Rift to track mouth movements. By having all the hardware mounted on the Rift itself, you don't have to worry about self-occlusion or trying to track facial features at shallow angles, but you do add somewhat to the weight of the Rift. You could do away with one of the eye-tracking cameras if you assume that one eye will always be pointing at the desired gaze target, and that the gaze target will always be the first object encountered by a ray projecting from that eye; and you don't care about anything other than the visual representation of the avatar (i.e. no interesting gaze-tracked UI elements, dynamic DoF or foveal rendering).
"edzieba" wrote:
You could have a three-camera solution: two eye-tracking cameras (nIR, internal to the HMD) to track gaze direction, and possibly eyebrow position, and one final camera mounted below the Rift to track mouth movements. By having all the hardware mounted on the Rift itself, you don't have to worry about self-occlusion or trying to track facial features at shallow angles, but you do add somewhat to the weight of the Rift. You could do away with one of the eye-tracking cameras if you assume that one eye will always be pointing at the desired gaze target, and that the gaze target will always be the first object encountered by a ray projecting from that eye; and you don't care about anything other than the visual representation of the avatar (i.e. no interesting gaze-tracked UI elements, dynamic DoF or foveal rendering).
Sounds pretty perfect to me. It's also sounding like weight may be an issue though. I suppose if these elements were ever to be added (Officially or by user modification) the benefits of the setup would have to be worth the extra weight and redesign effort.- I was going to talk here about how I run a site that has a lot of content that is applicable to this, but in researching my reply, I stumbled into this, which is probably way more relevant to the discussion:
http://www.grimace-project.net/about
It has Commercial and CC software licenses. - On a completely tangential note, since you seem to be talking mostly about facial expressions, it's worth mentioning that the Emotiv Epoc (an EEG based system) has among it's features a set of libraries that can read emotional states, and facial expressions from EEG data, rather than using a camera to figure it out.
While the cost of Epoc headsets is pretty high, and their development licensing is pretty confusing and expensive, it's still worth keeping in mind as an alternative.
After all, you seem to be struggling with reading facial expressions while wearing a device on your face that makes most of it impossible to see. XD
So... A device that can determine facial expressions without actually having to see your face does solve many of those issues.
However, as a final warning, I have heard that the EEG based method isn't nessesarily all that reliable.
It's certainly something I'll be testing if I can afford to...
