Up and Coming

Towards practical super-wide (180+) stereo video

26 replies
11917 views

Update:
Videos (you will need to have VR.js installed, preferable on Chrome, since the performance on Firefox is not as good):
Fondue party
Porch
Lab & crystal cove prototype
Robot soccer

I've been working on a stereo video panorama camera. It's essentially a ring camera array, with the cameras arranged as stereo pairs on the faces of a hexagon.

panocam_front_small.jpg

To do a full 360 degrees would take twelve cameras. Currently I'm only working with six cameras (three pairs), which gives about 240 degrees total horizontal FOV (120 degrees of head rotation + 120 degrees FOV in any particular viewpoint). Each camera is controlled by a Raspberry Pi, and then all the Pis are networked together-- this seems to be by far the most economical camera solution at only $70 per camera+pi (or $420 for all six). Since the Raspberry Pi cameras have a very low FOV, I've put a fisheye lens on each one (iPhone fisheye lenses, actually [~$6 each]), which bumps the horizontal FOV of each camera up to ~120.

panocam_overhead_small.jpg

Arranging the cameras in this kind of ring has a number of benefits:
1) The cameras can't see each other/can't occlude one another.
2) For any viewing direction (yaw) you only have to interpolate between two camera pairs, meaning that you only need the bandwidth to deal with four camera views at any time. This applies no matter how many cameras you have in the ring (e.g., you could have a ring with many more cameras to get higher quality and still only need to have the GPU/disk/network bandwidth to handle four at a time).
3) The more cameras you have the better the stereo quality. In the limit of an infinite number of cameras (a camera pair for every possible viewing yaw), the stereo would be perfect. This is in contrast to stereo over-under panoramas, where the stereo is fundamentally incorrect and increasing the panorama resolution will not fix it even in the limit.

The main limitation is that it only correctly supports yawing head rotations. Pitching your head will usually look fine, but strictly speaking the representation does not properly take into account the forward offset of your eyes. Rolling your head will cause problems.

Right now I have stereo panorama stills working, but I've only done a quick-and-dirty camera calibration, so there are distracting misalignments and seams between the cameras.

Edit: removed demos while I improve the calibration

Show previous replies

N

Nukemarine

Up and Coming

Thanks for the pdf link geekmaster. It goes into much better detail but meshes with what I was thinking. If they use 12 or more cameras for right eye and left eye mosaic I think the distortion is an acceptable tradeoff to using less cameras for a centerline mosaic.

Unless I'm reading it wrong, I think the paper's math is off a bit though. Due to the bio mechanics of turning a head on a neck, the eyes will follow a large radius than just the size of the head. I think the radius would be the back of the neck to the front of the eyes so perhaps 16 to 20 centimeters. The eyes will not be on the left and right side of a circle, but more parallel set apart on the average IPD.

Makes one wonder if Oculus is researching 360 stereoscopic a bit as well. I posted whether there are methods to take 360 degree snapshots in game and aside from 360 FOV Quake mod, nobody had an idea if such a thing existed yet. I'd argue this can be important for future VR game videos. Right now, we're locked on to seeing what the reviewer is looking at. Being able to create a 360 video (including stereoscopic 3D) means we can see game play video while being free to look around. Also useful for VR Machinima movies.

Thanks again.

P

PyryAuthor

Up and Coming

Yeah, you can use a more uniform arrangement of cameras. So if the cameras are labeled
A B C D E F
this setup uses AB, CD, and EF as pairs spaced 60 degrees apart, where the left eye only sees A,C,and E and the right only B,D, and F. But you could use AB as a pair, and then BC as a pair, and then CD, DF, and EF, where now every pair is rotated 30 degrees and each camera 15 degrees relative to its neighbors.

I've tested both arrangements, the paired arrangement and the uniform one, and they both work just fine. However, my preference is for the paired arrangement for a simple reason: it has fewer 'seams'. Every transition between cameras creates a visual seam (which can be somewhat mitigated* if you have depth or correspondence information between images). The uniform arrangement of cameras creates more seams, although those seams are less bad in terms of absolute pixel distances. That is, in the paired arrangement you'd see seams every 60 degrees, one from AB to CD, and another from CD to EF. In the uniform arrangement, you see seams every 30 degrees, one from AB to BC, then from BC to CD, and so on. In a full ring you'd see six seams for pairs, but twelve for uniform. Having tested both, my feeling is that perceptually, any seam is noticeable, so I think its better to have half as many seams, even if they are technically twice as bad.

* There's no free lunch here with depth information or correspondences: any sparse sampling of the full 4d lightfield is going to introduce artifacts of one variety or another. With depth information you can correct seams for opaque, lambertian surfaces. However, semi-transparent and non-lambertian (for example, very shiny) surfaces will still have issues. Semi-transparent surfaces are particularly difficulty, because they violate the assumption behind depth information that every pixel corresponds to one physical surface.

P

PyryAuthor

Up and Coming

"nukemarine" wrote:
I posted whether there are methods to take 360 degree snapshots in game and aside from 360 FOV Quake mod, nobody had an idea if such a thing existed yet. I'd argue this can be important for future VR game videos.

This type of camera ring will work perfectly well for game footage, actually even better than real life because you can arrange the cameras in a physically impossible way where they do not project forward at all. That kind of arrangement greatly reduces the seams between views, but is not possible in real life because the cameras would occlude one another.

N

Nukemarine

Up and Coming

"Pyry" wrote:
Yeah, you can use a more uniform arrangement of cameras. So if the cameras are labeled
A B C D E F
this setup uses AB, CD, and EF as pairs spaced 60 degrees apart, where the left eye only sees A,C,and E and the right only B,D, and F. But you could use AB as a pair, and then BC as a pair, and then CD, DF, and EF, where now every pair is rotated 30 degrees and each camera 15 degrees relative to its neighbors.

Pyry,

Thanks for the reply.

When using all 12 cameras (or even more), it wasn't about using them in pairs. Like with the article geekmaster linked, it's about creating two 360 panorama mosaics from all the cameras. To create the stereoscopic 3D, the mosaic grabs different the portion of the film that corresponds with the look angle for either eye.

That means some calculations must be done to determine the best portion of the image to grab. This could be based on the idea the head sweeps in an arc with a radius of about 18cm with eyes that are about 42 degrees apart on that arc. I'm no photo expert but I assume that for each camera view, you want to grab as close to centerline of the camera as possible for both eyes. I don't know if the software you use can accomplish that though, but it seems doable. The real benefit of this approach is it allows you in the future to create 360 cameras with more lenses make less distorted panoramas.

Anyway, I rewatched the lapdance video on VR Player. The settings were a 25 horizontal seperation (IPD) and a 2.0 barrel distortion. That reduced a lot of the warp when I turned my head. With the static areas of the room, the stitching was well done with some areas in the upper and lower portions more apparent. I did notice the stitching artifacts the most when the dancer walks by the camera which suggests the software couldn't blend as easily with moving objects. I assume this means that stitching is more noticeable with close up objects and not movement

Using more cameras does mean more blending artifacts. However, that seems a fair tradeoff for a less distorted panorama for each eye. If it's possible, try to film the same scene with variant set-ups then get the opinion others as to which looked best. Show it to people without telling them the method used and see which they prefer.

Sorry for the lengthy response. I just think that when VR becoming mainstream then 360 degree cameras including the 3D stereoscopic variants will become more and more popular. I'd like to see people grab that get this to grab market early.

R

relaxman

Participant

Great topic!
This device is my dream since years :)
I'm planning to do this with 6 (3x2) Gopro Black's.
I have 2 already.. Can this work?
Or the cheaper solution, with Mobius actioncam?

Now i try to make tests in calm nature with 2 cams,
rotating on a tripod. If i have the 3x2 videofile, what is the
easiest method for stitching to a single 360 video,
so anyone can view with vrplayer and the rift?

thanks

G

geekmaster

Up and Coming

There is more info here:
viewtopic.php?f=33&t=683&p=77976#p77976

Read the linked pages, and the pages that they link to. Those ideas (especially the stuff about stitching) can be used in video as well.

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

J

j1vvy

Up and Coming

I have been doing my own experiments into panoramic stereo video using multicamera rig.
http://photocreations.ca/3D/index.html

I have found that arranging the cameras in pairs was better than using a slice of each camera to make up each pair.
The seams that are created by stitching the images together can become very noticeable with video from a multicamera rig. By using pairs of images the seams for the left and right eye are in the same place so virtually eliminates half the seams.

Furthermore I have found that it is possible to interweave the camera pairs to minimize the parallax error that makes up each panorama by getting the lenses as close together as possible while still maintaining stereoscopic separation that is needed. So if the cameras are labeled A B C D E F G H I J, The pairs are AD, CF, EH, GJ, IB,

My problem is finding inexpensive cameras that have global shutter and genlock for timing. Between the rolling shutter and no sync the Mobius setup is best when left stationary.

I am thinking a pair of these cameras might work if good sync can be achieved. http://www.tamaggo.com/en

The camera has a 200° circular FoV. The cameras would be arranged to point ahead but slightly outwards, giving stereo vision straight ahead for about 170° and additional 30° peripheral vision for each eye. This way there should be no occluded view.

P

PyryAuthor

Up and Coming

I've posted some videos that are viewable online in the rift (see the first post for the update) if you have VR.js installed.

P

PyryAuthor

Up and Coming

j1vvy: I've noticed the same things. However, I think the better solution to the parallax seams is not to try to pre-stitch the videos together into one big panorama (which will unavoidably introduce distortion), but rather to estimate a depth-map (or some other representation of scene geometry, e.g., a mesh produced from multi-view geometry techniques) for each frame, and then use that depth map/geometry to do a view-dependent stitching between the cameras. This has the additional benefit of being able to accommodate some amount of head translation (i.e., as reported by the position-tracking of the crystal-cove prototype).

S

StephaneA

Participant

It seems that Jaunt doesn't shoot in stereo but rather rebuild the depth in post production from the camera array.
Does any body knows anything about depth extraction from multiple cameras ?
http://www.jauntvr.com/technology

Show more replies

Get Help

Sign up

Login with SSO

Login to the community

Login with SSO