Split Post Processing

You should PM leapmotion_alex over on the oculus reddit subforum.
he recently mention something about scaling up the image by 1.55 which cropping would effectivly do in this thread

http://www.reddit.com/r/oculus/comments/2puass/random_guys_leap_motion_impressions/

Responded on reddit, but wanted to keep the discussion going here as well. The 1.55 scaling doesn't apply to the images, but to world objects like the hand. This ensures that the virtual hand aligns with the real hand shown in the images -- necessary because the Leap cameras are separated by 4cm, while the average human eyespan is 6.4cm.

@alex
Thanks for the clarification. With artyom's oculus changes we will finally be able to scale the camera if we want the VR geometry to fit the passthrough. Still wondering what crop factor is needed on the corrected image for image passthrough to appear correct if anyone of your technical guys could get back to us on that, it would be very useful!

@opamp
Your cropping code almost worked out of the box, thanks for that! Had to fiddle with it for a bit so it would look right for the 200% zoom factor. I came up with this


(open image to see full graph, oculus should really resize images in their forums...)

It uses the rendered scene as a final blend, which allows for intensity parameter based blending of the passthrough effect. The 0.25/0.75 as center points for each image is only valid for the 0.5/0.5 crop factor, unsure how to make a formula so that the center of the images always scale correctly (I assume that is what your function was trying to do, but TexCoord as UVs may have messed with that since we modify it to the 2 images).


Then there is a separate problem with the way the distortion is calculated in the plugin at the moment, e.g.



The left image is the distortion used for the lookup and the right is a regular grid texture warped. It seems to bias the left/bottom corner.

I wonder if this is due to downscaling the distortion from 32bit to 8bit (couldn't get 32bit per channel textures to work in UE), but I think it is much more likely due to scaling the float range to 0-255 when some values are above 1.0 in the float range and these are supposed to be dropped. Will adjust the plugin code and see if I can't get this to work right.

After some pointers from epic and leap I finally got the 8bit passthrough to look alright!

Apparently textures by default use gamma correction which is used to widen perceived colors. This applies a power curve on your values which is why I got the UV center skew earlier. Simply calling

utexture2DPointer->SRGB = 0;

will turn this off.

With that correction done the crop was based on the oculus DK2 projection matrix (@artyom btw if we have access to this as a raw value, shaders which depend hmd FOVs would be adaptable to any newer hmds with different FOVs e.g. DK1 vs DK2)



which gives

H = 0.25/0.929789 = 26.89%
V = 0.25/0.752283 = 33.23%

where the percentages are the percent of the pre-warp image that should be retained around the image centers for each axis.

With all that done, the 8bit per channel passthrough was ready to use. Below is an example of it applied for a uniform grid source (NB: open these images in a separate tab to see them fully, some may be cropped due to how the oculus forum works)



Some squiggles in the lines are apparent and these are likely due to the 8bit downsampling, more on this later. At this stage I was curious of the performance impact of this setup, so I ran both the GPU and CPU profilers during rendering.


The GPU profiler showed that the LeapPassthrough shader took 0.19ms per eye to complete, a very tiny and acceptable fraction of the total post process cost.


From the CPU profiler you can see two timings 2.54ms and 1.77ms they correspond to the total time the CPU spent on the plugin tick, out of which most is spent on copying image and distortion pointers. Initially I used a CPU function that culled outside ranges and inverted values. Simply by swapping the channel inversion to the shader and ignoring values outside the range (they won't be shown anyway due to cropping) we saved about 0.8ms which is more than the entire GPU cost of this post process.

The final material (UE visual shader) ended up looking like this



with the always horizontally centered cropping material function given by


Which is 66 instructions for the optimized version. Since we're already in the post process I went ahead and tried a simple depth blend.



It's like replacing your office space with an open roof, it had the feeling of a future AR style office, quite a few interesting things you can do from here.

The 8bit distortion map still suffers from line warping which causes some of the pixels to align incorrectly when using the HMD. The next step now is to get that 32bits of distortion per channel working and maybe finally this will be incorporated into the plugin :smile:

If you know anything about float channel textures in UE, drop me a line, formats are still pretty poorly documented :(

Finally got the final 32bit per channel texture version working. Thanks a lot for your help opamp!

you can now try it by grabbing the plugin update and setting your character to the convenience content one included in the plugin



and then you can transition from VR to AR with a simple gesture