Increasing Immersion of 360 Panoramas

[kojack]

While some consider them "not VR", viewing 360x180 panoramas on a VR headset is quite cool. Consumer 360 cameras are cheap and software like Hugin and PTGui let you turn normal photographs into panoramas.
There is a problem with panoramas though: depth.

Assuming you are using only one eye to view a panorama in VR, the image will be correct in all directions. Of course there won't be accomodation (focus), consumer VR headsets don't do that yet. The problem comes in when you have stereo vision.
Look around in a panorama using both eyes, everything looks ok. It's fairly flat, but ok. Now look at the ground. Oh crap, I'm flying in the air!
The standard way a panorama is rendered is by projecting it onto the inside of a sphere. This sphere is probably around 30m radius (that's where our stereo depth perception breaks down). This makes things around us look far away, which tends to suit most panoramas. But it also means the ground below us is 30m away! With stereo vision, we feel like we are way up in the air, even though the panorama was captured at head height.

One way to fix this is to use 3d panoramas (stereo images). This captures depth for the entire view. But these have issues too: you can't tilt your head sideways (otherwise your eye alignment won't match the panorama) and it's harder/more expensive to capture.

The technique I'm using in this demo is to add in ground plane geometry and project the panorama onto it, in addition to the usual sky sphere. From a fixed 3DOF view, a ground plane with correct projection will look the same, except you can see the depth is correct. You can tilt your head, the depth is dynamic instead of baked.

This ground plane does magnify another issue with panoramas: they are 3 Degrees Of Freedom only. Well, they don't have to be. In this demo you can change to a 6DOF mode. You can now walk around the panorama. Of course this distorts the view quite a bit. In particular any vertical surfaces projected onto the ground plane look distorted when your head isn't in the exact centre of the panorama space. But it still looks cool (especially if you are looking down, since the nearby floor is less distorted).

I've also added a 4DOF mode. In this mode, you can look around (3DOF), but also move up and down. Horizontal movement is disabled. You can bend down and get a closer look at the ground, with less distortion than the full 6DOF mode.


The demo includes 4 panoramas. Three are from the European Southern Observatory (their website has many awesome panoramas at crazy resolutions). The other is a Nvidia Ansel capture from the game The Witness.
Controls:
A - Toggle ground plane
B - Cycle to the next panorama
X - Cycle to the next degree of freedom. There are 3, 4 and 6 DOF available.
Right Thumbstick click - Reset position/orientation
Index Triggers - Spawn a cube
Grip Triggers - Grab a cube

No, I didn't add cube juggling just for the demo. This was built on a base project for my students, so it already had some hand/physics stuff in there. 🙂

Demo:

Rift/Rift-S version. Would probably work on SteamVR with a Vive too, but is untested.

https://github.com/rajetic/DepthPanorama/releases/download/v0.1/Depth.Panorama.Test.v0.1.Rift.zip

Quest version. You can upload it using Side Quest

https://github.com/rajetic/DepthPanorama/releases/download/v0.1/Depth.Panorama.Test.v0.1.Quest.zip

How does it feel? Which degree of freedom mode do you like best? Did you ignore the panoramas and just juggle cubes?

[kojack]

As long as you know the original camera height and the sky sphere is larger, you can place the floor where ever. But for a drone, probably not as useful. 🙂

Although... if you had a log of the drone's altitude, you could make the floor plane match, so as the drone goes low (or lands) the ground will flatten the sphere at the correct position. 🙂

One day I'll come back to that depth generating technique I mentioned above that required user editing of the panorama. It was interesting, but editing the panoramas was time consuming.

[XLRLimits]

Hello this looks very promising. I have just now stumbled upon this, i was also interested how i can make more realisting environments with 360 panorama and projecting on to a plane the bottom i think helps a lot.

Using the shader you posted works but does not do the trick for me. It is not quite clear how you were able to make it work so well on the device. Do you think you can share more details like how big the sphere is, how big the plane is, does this work only  for 1 camera, maybe even a github project?

[kojack]

I'll see if I can knock together a quick demo (with source) tomorrow.

I had the sphere as around 20m (iirc) and placed the floor at -1.7m. It works with stereo rendering.

The shader requires the camera to be placed at 0,0,0, so that's why the floor is at a negative. That would be easy to offset though.

[RoastPork]

Can't wait to see! This is gonna be wild.

[kojack]

I got distracted porting System Shock 2 to Bonelab. 🙂

But I've got a demo working. It supports Rift and Quest and uses UltimateXR (for the avatar).

I just need to trim down the bloat (Oculus SDK and UltimateXR combine to over 1.4GB, a lot can be cut) and track down the licensing of the sample panoramas I used. Then I'll post a repo.

Correction to the above: my testing today shows a sphere size of 150m makes the sphere to ground transition much less obvious. But that does require large outdoor areas to avoid distortion.

[kojack]

Ok, got it up.

Here's a full repo of a Unity 2012.3.5f1 project showing the technique.

Binaries of Quest and Rift demo builds are there too. (Tested on Quest 2 and Quest 2 Airlink)

Yeah, the build got a bit big, I think Unity expanded all the panoramas to raw and they are 8K.

The really useful bits are just the shader file (already posted) and a simple .cs file to cycle and turn the floor on/off.

I had to add a timed credits panel. 3 of the 6 panoramas I used as samples are from the European Southern Observatory. Their website has a huge collection of insane resolution 360 panoramas of radio telescopes. But their licensing requires that a credit line must appear directly connected to the display of the panorama (not in a separate text file, etc). So I just make a 4 second panel appear in VR with the credit.

Press A on the right touch controller to cycle through the 6 provided panoramas. Press B to toggle the floor effect on/off (the whole point of this).

Or maybe it was B to cycle/A to toggle. I've already forgotten. 🙂

I stripped out the UltimateXR stuff, it was causing problems. So no hands. I also removed around 400MB from the Oculus SDK (sample stuff) to get the size down.

https://github.com/kojackdev/PanoramaTest

(This is much more simplified than the demos from the first post that didn't come with source. There's no floor tilting, 3dof and 4dof modes, custom floor heights saved per panorama, etc)

[relaxculus]

Cool, can i test it with my own images? Maybe copy to some folder?

[kojack]

At the moment it only shows ones set up in Unity on the PanoramaController object (there's a texture list there). So you'll need to rebuild it to see anything different.

I could probably add in general searching in a folder, I just don't want to deal with user interface stuff. 🙂

[relaxculus]

yeah, just list the app picture folder is fine 🙂

[kojack]

A present for you. 🙂

There's a binary on the github called PanoramaTest_Quest_BYO.apk. It looks in the Quest's Pictures folder for jpgs and pngs and uses them. No built in ones are provided (which did help shrink the apk from 460MB to 25MB).

It reloads a panorama every time you show it, that has a delay (they can be big), but it won't take up more memory than a single pano at a time.

Hopefully it works, I hate dealing with android manifests and permissions and crap. It took a while just to get file loading from Pictures to not fail.