No way to add focus information to individual pixels

"antigravity" wrote:
The super fast eye-tracking with realtime DOF changes has got to be the ultimate solution, but if someone did a demo where realtime DOF was intelligently based on where your (0,0) HUD target intersected with your visuals? I'd bet you'd nail 80% of the effect and remain somewhat practical.

Based on what Tom says about attempting this, it seems like the realtime eye-tracking approach is going to be the better idea. Had anyone attempted to mod a rift by adding internal cameras for eye tracking?

"tomf" wrote:

"antigravity" wrote:
The super fast eye-tracking with realtime DOF changes has got to be the ultimate solution, but if someone did a demo where realtime DOF was intelligently based on where your (0,0) HUD target intersected with your visuals? I'd bet you'd nail 80% of the effect and remain somewhat practical.

People have tried this - it's pretty horrible. They've also tried setting the focus to where your mouse/reticule/aim is - that's bad as well. The problem is that your eyes move before either of these sorts of motions, and usually only by a little bit. So what happens is your eyes move, they see blur, try to focus (but of course they can't - it's not that sort of blur), then your aim/view shifts half a second later, and the blur goes away, and now your eyes refocus. It's quite straining on the eyes.

Good point.. when you take into account that physical eye-strain element there's just no way in hell it would be worth it. Half of my face is bloody sore just thinking about it.

I guess at the very least as a developer we can still do selective DOF on shots where you safely assume player focus or want to control it. (cut screens with dialogue, specific object of interest). Works well for Pixar films and haven't actually seen it on the Rift yet.

ag

Interesting. So I conclude that the problem is that we have no means (that I am aware of) to read out the focus distance from the eyes. If we could, the rendering can be adjusted to it.

I think I agree with tomf on this. The eyes are normally in control of focus and will naturally and quickly try to compensate for out-of-focus views. If you use sub-optimal ways to detect focus your eye-focus will start a battle with that method, which is very tiresome.

In the Pixar/movie situation you are already use to the situation where your eyes are not in control: the movie is directing where you look, what needs to have focus etc. In VR, immersion is improved by the feeling that you have full and natural control over the environment. Therefore I do not think that having the engine setting focus will work well, not even in cut-scenes.

"serrarens" wrote:
Interesting. So I conclude that the problem is that we have no means (that I am aware of) to read out the focus distance from the eyes. If we could, the rendering can be adjusted to it.

I'm not certain of the state of the art in eye tracking, but I'm pretty sure it's physically possible to determine the focus based on the current curvature of the surface of the eye. Doing so inside the headset with the limited space for LED light sources and pickup cameras might be impractical though.

"jherico" wrote:
I'm not certain of the state of the art in eye tracking, but I'm pretty sure it's physically possible to determine the focus based on the current curvature of the surface of the eye. Doing so inside the headset with the limited space for LED light sources and pickup cameras might be impractical though.

Focus is usually determined by using a half-silvered mirror (beam splitter) and direct visual inspection, or with a camera, to observe (or autofocus) an image projected onto the retina.

Observing a projected image focused on the retina is much more accurate than observing eye lens shape. Perhaps an infrared image could be used, but it may be necessary to adjust for chromatic differences.

Here is an interesting patent that gives details on how to use a similar method for eye tracking:
http://www.google.com/patents/US7298414

Perhaps something similar (that does not violate any patent) could be used in the Rift.

A keratometers (ophthalmometer) for measuring corneal curvature may use a similar (or other) method, but they tend to be rather large:

http://www.optivision2020.com/keratometer.html
http://www.nao.org/cec/Online%20CECs/CEC%20pdfs/Keratometer%20online%20text.pdf

The eye-tracking patent mentioned above uses something more like a digicam autofocus mechanism (which can be very compact).

"jherico" wrote:
I'm not certain of the state of the art in eye tracking, but I'm pretty sure it's physically possible to determine the focus based on the current curvature of the surface of the eye.

I don't know anybody that actually looks at the lens focus. Instead they track each eye separately and look at where they converge. That tells you how far away the object the person is looking at. Then they change the focus of the display to be at that distance. It's a rather bass-ackwards way of doing it, but it seems to work OK in practice.

"tomf" wrote:

"jherico" wrote:
I'm not certain of the state of the art in eye tracking, but I'm pretty sure it's physically possible to determine the focus based on the current curvature of the surface of the eye.

I don't know anybody that actually looks at the lens focus. Instead they track each eye separately and look at where they converge. That tells you how far away the object the person is looking at. Then they change the focus of the display to be at that distance. It's a rather bass-ackwards way of doing it, but it seems to work OK in practice.

That is virtual focus, which is applicable to VR. Measuring lens curvature as suggested only measures focus to the screen, mostly useful for determining eyeglass prescription (or eye-tracking as shown in that patent I linked), and not the virtual content.
But yes, determining what pixel you are gazing at will tell the software how far away the virtual object is in VR. Measuring eye focus might be a useful feedback mechanism to servo autofocus eyeglasses (even when computer controlled for VR).

The problem with variable focus is that the eyes are reported to move up to 700 miles per hour, and eye-tracking plus lens fluid adjustments cannot yet track at that speed, with the latency probably causing noticeable eye strain.

"tomf" wrote:
I don't know anybody that actually looks at the lens focus. Instead they track each eye separately and look at where they converge. That tells you how far away the object the person is looking at. Then they change the focus of the display to be at that distance. It's a rather bass-ackwards way of doing it, but it seems to work OK in practice.

Yeah, I didn't really think it through. Obviously if a user's focal distance is anything other than the fixed distance to the LCD while wearing the Rift, they're just going to be blurring the screen. The convergence issues brings me back to one of my major gripe with the SDK, that there's no standardized mechanism for dealing with people who's eyes don't converge in the conventional manner. Since I'm one of those people, it's way more important to me than getting the IPD right, since it has much more of an impact on my ability to fuse images and perceive something in 3D.

"jherico" wrote:
Obviously if a user's focal distance is anything other than the fixed distance to the LCD while wearing the Rift, they're just going to be blurring the screen.

The lenses produce what is called "collimated" light, where all the light rays from a particular pixel are parallel to each other. It's a very strange concept, but it means that the focus for the panel is at infinity. It is certainly not 50mm from your face - you can't focus that close! It is literally "at infinity" - the light rays are parallel. So yes, it's a fixed distance, but that distance is infinity - your eyes are focused the same way they are for mountains, clouds, and stars. It's quite relaxing for most people. And it's why people who are long-sighted don't need to wear their glasses in the Rift.

"tomf" wrote:

The lenses produce what is called "collimated" light, where all the light rays from a particular pixel are parallel to each other. It's a very strange concept, but it means that the focus for the panel is at infinity. It is certainly not 50mm from your face - you can't focus that close! It is literally "at infinity" - the light rays are parallel. So yes, it's a fixed distance, but that distance is infinity - your eyes are focused the same way they are for mountains, clouds, and stars. It's quite relaxing for most people. And it's why people who are long-sighted don't need to wear their glasses in the Rift.

Hi Tom,

When working with stereoscopic screens, I found that I had to set the convergence distance (which induces frustum asymmetry) to be pretty close to where in our simulator's content I wanted to look. That is, if I was looking at the controls 0.5 meters in front of me in the cockpit, convergence had to be something like 1-2 meters at most. The result was that when I would look out of the airplane's window at a building 1000 meters away, my eyes felt like they wanted to pop out of my head and punch me. :-(

My question is: with collimated lenses and horizontally separated displays (which provide a comfortable 'infinite' viewing distance with an infinitely far convergence/no asymmetry), what happens if the user wants to "focus" on a detail that is close to the viewer's head in virtual space?

Is the actual user experience to simply see everything in focus with no eye movement? Or do the user's eye directions converge while their lenses don't refocus?

Thanks!!
Ben