No way to add focus information to individual pixels

I have seen the future and it is Siggraph.




Nvidia is here demoing their light field displays. They are able to include focal information across the entire view because it is a light field. In addition they can change eyeglass prescription in software (by changing the arrangement of the images through each microlens). It's magical! The biggest drawbacks are that it slashes resolution and can be a little tricky to render with focal information at good frame rates (easy for raytracing, hard for rasterization, tricky but workable for rasterization+depth maps; think VorpX).

And the presenter talked very fondly of the oculus rift :)

That video is awesome. Now were can I get some cheap microlens arrays?

Google found this:

"

How to make your own Microlense Array [DIY LightField Camera]
Making your own DIY LightField Camera is not very hard, provided that you have a DSLR camera with detachable CCD back.
One of the things that may be hard to get is an array of microlenses – e.g. the cosine film mask we mentioned in our earlier Do it Yourself post is virtually impossible to come by these days.
Microlense sheets are available at reasonable prices from a few sources, but their specifications may not be ideal for your camera or image sensor (e.g. size, focal length, …).

That doesn’t put off real LightField fans, though: Mats Wernersson shows you how to create a microlense array yourself.

He basically uses a copper plate with little, defined dents as a mould for his custom polycarbonate lens array. After polishing the metal plate, the little indentations are created in a regular pattern using a modified Sherline mill. Using this matrix, he uses a small home-made molding press and a sheet of polycarbonate.

The final result of his DIY project looks impressive: It’s an array of about 6000 microlenses on a 36 x 48 mm surface, with a focal length of 1.36 mm and an f-number of 2.8.



Mats provides a detailled description of this project over at CameraMaker.se, including a video of his customized Sherline mill in action.

Be Sociable, Share!

Which leads to this:

"
The idea is to place a ball bearing on a copper plate and smash it with a hammer. The ball bearing leaves a dent that will be used as a mold for a lens. The process just needs to be refined a bit.
...
Small spherically shaped dents are created by hammering the little ball bearing on the copper plate. This is done in a Sherline mill. The mill column was removed and replaced by a beam. The beam is lifted by a cam mounted on a stepper motor. The ball bearing is fixed at the other end of the beam.
...
The final result looks good. Now it is time to do some light field experiments. The result can be viewed here: Plenoptic Camera. ...

Although I want such a microlens array to use in an HMD, this DIY microlens array was tested in a plenoptic (light field) camera:

"
Refocusing


Changing perspective


The unprocessed light field image
...

So now we know that we can build a DIY super-lightweight light-field HMD with DIY microlens arrays, similar to what was featured in that SIGGRAPH video from Nvidia.

In essence, many partially-overlapped images provide the "per pixel cluster" focus information we need to provide an even more highly-awesomized VR experience! And with the microlenses too close to focus on the lenses themselves, we have virtually the same perceptual effect as per-pixel lenses and per-pixel focus information.

I want this ASAP, so this is another project on my "To Do" list. My friend's son just got a nice mill, and they want to to use it too, so this sounds like a cool first project. :D

His fab process is pretty clever! My dad has a mill, hmm...

This is awesome too:

I posted more about NVIDIA Research on hNear-Eye Light Field Display with Lens Array Optics in this other thread:

viewtopic.php?f=33&t=3015

"Harley" wrote:
NVIDIA Near-Eye Light Field Display with Lens Array Optics?

NVIDIA Research are showing of a very promising Near-Eye Light Field Display prototype HMD at SIGGRAPH this year:

http://www.engadget.com/2013/07/24/nvidia-research-near-eye-light-field-display-prototype/

These Near-Eye Light Field Displays features Thin Magnifying Lens Array Optics that enables the human eye to do depth focus and defocus at different depths for the same image, and this type of setup also allow them correct for someone's glasses or contacts lens prescription in software to enable sharp focus, which are two things that the LEEP optics in the current Oculus Rift DevKit do not offer.

http://www.youtube.com/watch?v=deI1IzbveEQ



Note that the prototype in these images and videos is just a rapid prototyped model that is basically a stripped Sony HMZ-T2 with everything but the OLED displays and controller boards removed, replaced with NVIDIA Research's own Thin Magnifying Lens Array Optics for Near-Eye Light Field Displays in a 3D printed HMD glasses shell and the unique software rendering components to use a such lens array.

https://research.nvidia.com/publication/near-eye-light-field-displays
http://research.nvidia.com/sites/default/files/publications/neld-abstract.pdf

Near-Eye Light Field Displays

Research Area: Stereoscopic 3D
Author(s): Douglas Lanman (NVIDIA), David Luebke (NVIDIA)
Date: July 2013

This light-field-based approach to near-eye display allows for dramatically thinner and lighter head-mounted displays capable of depicting accurate accommodation, convergence, and binocular-disparity depth cues. The near-eye light-field displays depict sharp images from out-of-focus display elements by synthesizing light fields that correspond to virtual scenes located within the viewer's natural accommodation range. While sharing similarities with existing integral imaging displays and microlens-based light-field cameras, the displays optimize performance in the context of near-eye viewing. Near-eye light-field displays support continuous accommodation of the eye throughout a finite depth of field; as a result, binocular configurations provide a means to address the accommodation-convergence conflict that occurs with existing stereoscopic displays. This demonstration features a binocular prototype and a GPU-accelerated stereoscopic light field renderer.

I think that it is optics engineers of this quality that Oculus VR should aspire to employ:
https://research.nvidia.com/users/douglas-lanman
https://research.nvidia.com/users/david-luebke

I did some math earlier today and figured that the effective resolution of nVidia's solution is something in the ballpark of 256x144 using a 1280x720 panel (downside of using microdisplays, super high pixel density, not so much on the total resolution). At least, that's the per-angle resolution.

So, if you can imagine your whole field of view being a 256x144 image, and your videocard having to render tens or hundreds of different camera angles, you can see the issue. This might be where the future of HMDs lies, but it seems like the distant (decades) future to me.

"bensupnik" wrote:
My question is: with collimated lenses and horizontally separated displays (which provide a comfortable 'infinite' viewing distance with an infinitely far convergence/no asymmetry), what happens if the user wants to "focus" on a detail that is close to the viewer's head in virtual space?

Is the actual user experience to simply see everything in focus with no eye movement? Or do the user's eye directions converge while their lenses don't refocus?

This last sentence is what actually happens - they get depth information from vergence, but not focus. The fact that the lenses of their eyes are still focused at infinity doesn't seem to bother people much, if at all. (by which I mean I've never actually met anyone who got eye strain from NOT having to refocus their eyes!)

Actually there is two way to solve the initial problem :


1:
_Check the retina with captors/camera, to detect when our eye is trying to focus on something.
_calculate the new image according to the detection of the retina, to adjust the blurry/clear distance dynamically.
_Send the new image to the classical screen



2:
Dont detect anything from the eye
_Calculate not only one image (One static state by pixel), but a "luminous ray" information
_Find a technology of screen, who send this luminous information to the eye, all of it, per pixel, and not only one "static color".
(So the eyes perceive, from a direction/pixel, the same "light information", as in reality, and is capable to focus/unfocus by itself.)


I hope you get the general idea of what i'm saying, my english is not that good ;)