Achieving natural depth of field

You can look at something directly in front of you, then look past it into the background without really moving your eyes much. Eye tracking wouldn't catch this.

It would! Or at least in theory. Even though it would physically be on the same z-plane as everything else on screen, the point of convergence would be different (you cross your eyes more/less).

The options are not affordable with current technology just yet, but they will be one day.

I have family that develop Eye-Tracking and this might not be an issue anymore. I'm trying to get a hold of someone at Oculus VR to talk about it.

Checkout NVIDIA Research on Near-Eye Light Field Display with Lens Array Optics posted in this other thread:

viewtopic.php?f=33&t=3015

"Harley" wrote:
NVIDIA Near-Eye Light Field Display with Lens Array Optics?

NVIDIA Research are showing of a very promising Near-Eye Light Field Display prototype HMD at SIGGRAPH this year:

http://www.engadget.com/2013/07/24/nvidia-research-near-eye-light-field-display-prototype/

These Near-Eye Light Field Displays features Thin Magnifying Lens Array Optics that enables the human eye to do depth focus and defocus at different depths for the same image, and this type of setup also allow them correct for someone's glasses or contacts lens prescription in software to enable sharp focus, which are two things that the LEEP optics in the current Oculus Rift DevKit do not offer.

http://www.youtube.com/watch?v=deI1IzbveEQ



Note that the prototype in these images and videos is just a rapid prototyped model that is basically a stripped Sony HMZ-T2 with everything but the OLED displays and controller boards removed, replaced with NVIDIA Research's own Thin Magnifying Lens Array Optics for Near-Eye Light Field Displays in a 3D printed HMD glasses shell and the unique software rendering components to use a such lens array.

https://research.nvidia.com/publication/near-eye-light-field-displays
http://research.nvidia.com/sites/default/files/publications/neld-abstract.pdf

Near-Eye Light Field Displays

Research Area: Stereoscopic 3D
Author(s): Douglas Lanman (NVIDIA), David Luebke (NVIDIA)
Date: July 2013

This light-field-based approach to near-eye display allows for dramatically thinner and lighter head-mounted displays capable of depicting accurate accommodation, convergence, and binocular-disparity depth cues. The near-eye light-field displays depict sharp images from out-of-focus display elements by synthesizing light fields that correspond to virtual scenes located within the viewer's natural accommodation range. While sharing similarities with existing integral imaging displays and microlens-based light-field cameras, the displays optimize performance in the context of near-eye viewing. Near-eye light-field displays support continuous accommodation of the eye throughout a finite depth of field; as a result, binocular configurations provide a means to address the accommodation-convergence conflict that occurs with existing stereoscopic displays. This demonstration features a binocular prototype and a GPU-accelerated stereoscopic light field renderer.

I think that it is optics engineers of this quality that Oculus VR should aspire to employ:
https://research.nvidia.com/users/douglas-lanman
https://research.nvidia.com/users/david-luebke

NICE!!

Ok, back from the excitement...

I remember when I was first introduced to the Lytro and did some research into the technique. My gut told me that the results were reversible: that one could emit instead of just receive. It's good to know that someone with the tools to accomplish the task was able to start the ball rolling! (As has been stated before, ideas abound - practical application is where reality exists.)

Doing some quick trig sketches (helped along by my favorite solver: SolidWorks - no kill like overkill!) I found some interesting numbers. I endeavored to determine what the rough pixel density would have to be to achieve various results on such a near-eye setup. Here are my results:
Virtual image distance: 8 inches (203mm) from eye.
Wanted virtual pixel density: 300 PPI.
Eye to HMD display distance: 0.5 inch (13mm)
Resulting PPI on HMD: 4808 PPI.

Virtual image distance: 8 inches (203mm) from eye.
Wanted virtual pixel density: 300 PPI.
Eye to HMD display distance: 1.0 inch (25mm)
Resulting PPI on HMD: 2398 PPI.

Virtual image distance: 8 inches (203mm) from eye.
Wanted virtual pixel density: 100 PPI.
Eye to HMD display distance: 0.5 inch (13mm)
Resulting PPI on HMD: 1600 PPI.

Virtual image distance: 8 inches (203mm) from eye.
Wanted virtual pixel density: 100 PPI.
Eye to HMD display distance: 1.0 inch (25mm)
Resulting PPI on HMD: 800 PPI.

Note that what I'm terming "virtual image" is what you would get if you were to project the pixels being displayed on the HMD out to a matching surface at a given distance from the eye. The goal is that if I were to display a virtual computer screen in my "game world" and look at it like I would in real life, the virtual computer screen should have the same "pixel" (texel?) density as the equivalent screen in real life.

Doing some quick research on the smallest LED displays, especially the OLED, I found that the feature sizes being discussed were all at the 50 to 600 um size range. A wide range, but now I wanted to know what my estimated maximum feature size would need to be for a grouping of 3 RGB round dots at the PPIs listed above. This estimate is large due to not accounting for gaps or extra components needed in making a functional display:
4808 PPI: 2.45um
2398 PPI: 4.92um
1600 PPI: 7.37um
800 PPI: 14.74um

Even if said feature size was attainable, the really awesome methods being linked to above would result in even less virtual (aka perceived) pixel density - or would require even finer feature sizes!

I wonder if raster isn't the best technique for displaying information in an HMD... Got to explore that thought.

(2/10)

Could It be possible to (assuming you could track the eyes rapidly) find the object on screen in direct focus of the eye, adjust the FOV blur accordingly, then apply a sort of shader driven focal shift on the object in focus to correct for the lens in the eye, based on its proposed distance? I'm unsure how feasible this would be to do because I know very little about how shaders actually work, or whether or not rapid eye tracking technology is even something you can do affordably, but this is the only way I could think of that would adjust for the lens in the eyes. Tell me if you like the idea, or if I'm being dumb in some way. :)

are you talking about panums field now? if you focus on a near point and try to look behind that point you get double vision..

it is because every point on your retina corresponds to another point in the other eye..so if those points dont get activated at the same time your brain will not be able to fuse the images therefore you will see double beyond focus point...i can explain this for hours but my english is not the best and it will take some time to write here...this should not be a problem in the oculus since the screen should be at the focus point..if you had a screen behind the screen i could see this being a problem

I'm strongly disagree with software blur depth of field based on user focus. With only slight increase of realism, it will seriously tamper the picture. And also, it doesn't look as real in VR as it looks in the 2d monitor - your eyes are still getting converged image, just blurred in software.

However, it is generally safe and encourageable to forcibly blur objects that are closer to the viewer than 3" - human eye is physically incapable of focusing at this distances. Even if personally you can focus at as short as 2", it will start to cause some eye strain even at 5" and down 3" it becomes really bothering, so presumably, you won't be doing this.

The only real option here is light field projection, somehow. Microlenses are obviously not an option - all-around resource hog, and all for a slight divergence of incoming light, there should be a better way.

How about displaying your next venture with state of the art 3 dimensional hologram displays instead of those boring banners? With hologram display, you can add a life to your product exhibit with these 3 dimensional displays. With Olomagic, this confusing tricky technology has taken a whole new turn. It is easy to install, use, and change the images in Olomagic. Our team can help you to install it wherever you want to in just a couple of minutes. With USB drives, you can insert pictures and details to show it in 3 dimensional aspects.

Except that it's ungodly hard to generate a hologram because you literally must render it from every possible viewing position.

Hello, this issue is the one thing I immediately noticed when first seeing 3D cinema. Even with a camera or high quality rendering the dof won't be accurate since everybody looks at a different spot in every situation.
Since our brain 'scans' the image with help of the fovea the image will be out of focus in a simulated dof effect. And it gets tiresome for the eye to try to focus on something which it can't (because the projection is blurry).

So unless we really have eye tracking dof is kind of useless apart from the more obvious examples.

Providing the eye with something else than an infinite point of focus could be fairly easy with new hardware.
Just put piezo elements in-between the lenses and whatever they are mounted on. Since the focal distance of our lenses is fairly short it might be sufficient to modulate the voltage the piezos get. I haven't done the calculation if that is sufficient to account for a large enough amount of variable focus.
There are some products using piezoelectric actuators in camera auto focus so I think it might be a thing.

What's next our eyes are pretty much always focused on the area where the fovea is pointed at and - given the eye tracking is good enough to account for the fastest and minuscule eye movements we might not need to account for the focus at all, we might not be able to tell a difference.

I've got a tech demo up on share that has natural depth of field in it for live action capture. You need to follow the ball all the way to the end. When the ball is in your face, focus on the center of the ball, then focus on the background. You will notice the natural blurring of background and foreground appropriately.

https://share.oculus.com/app/live-action-360-3d-tech-demo