Forum Discussion
Towards practical super-wide (180+) stereo video
Update:
Videos (you will need to have VR.js installed, preferable on Chrome, since the performance on Firefox is not as good):
Fondue party
Porch
Lab & crystal cove prototype
Robot soccer
I've been working on a stereo video panorama camera. It's essentially a ring camera array, with the cameras arranged as stereo pairs on the faces of a hexagon.
panocam_front_small.jpg
To do a full 360 degrees would take twelve cameras. Currently I'm only working with six cameras (three pairs), which gives about 240 degrees total horizontal FOV (120 degrees of head rotation + 120 degrees FOV in any particular viewpoint). Each camera is controlled by a Raspberry Pi, and then all the Pis are networked together-- this seems to be by far the most economical camera solution at only $70 per camera+pi (or $420 for all six). Since the Raspberry Pi cameras have a very low FOV, I've put a fisheye lens on each one (iPhone fisheye lenses, actually [~$6 each]), which bumps the horizontal FOV of each camera up to ~120.
panocam_overhead_small.jpg
Arranging the cameras in this kind of ring has a number of benefits:
1) The cameras can't see each other/can't occlude one another.
2) For any viewing direction (yaw) you only have to interpolate between two camera pairs, meaning that you only need the bandwidth to deal with four camera views at any time. This applies no matter how many cameras you have in the ring (e.g., you could have a ring with many more cameras to get higher quality and still only need to have the GPU/disk/network bandwidth to handle four at a time).
3) The more cameras you have the better the stereo quality. In the limit of an infinite number of cameras (a camera pair for every possible viewing yaw), the stereo would be perfect. This is in contrast to stereo over-under panoramas, where the stereo is fundamentally incorrect and increasing the panorama resolution will not fix it even in the limit.
The main limitation is that it only correctly supports yawing head rotations. Pitching your head will usually look fine, but strictly speaking the representation does not properly take into account the forward offset of your eyes. Rolling your head will cause problems.
Right now I have stereo panorama stills working, but I've only done a quick-and-dirty camera calibration, so there are distracting misalignments and seams between the cameras.
Edit: removed demos while I improve the calibration
Videos (you will need to have VR.js installed, preferable on Chrome, since the performance on Firefox is not as good):
Fondue party
Porch
Lab & crystal cove prototype
Robot soccer
I've been working on a stereo video panorama camera. It's essentially a ring camera array, with the cameras arranged as stereo pairs on the faces of a hexagon.
panocam_front_small.jpg
To do a full 360 degrees would take twelve cameras. Currently I'm only working with six cameras (three pairs), which gives about 240 degrees total horizontal FOV (120 degrees of head rotation + 120 degrees FOV in any particular viewpoint). Each camera is controlled by a Raspberry Pi, and then all the Pis are networked together-- this seems to be by far the most economical camera solution at only $70 per camera+pi (or $420 for all six). Since the Raspberry Pi cameras have a very low FOV, I've put a fisheye lens on each one (iPhone fisheye lenses, actually [~$6 each]), which bumps the horizontal FOV of each camera up to ~120.
panocam_overhead_small.jpg
Arranging the cameras in this kind of ring has a number of benefits:
1) The cameras can't see each other/can't occlude one another.
2) For any viewing direction (yaw) you only have to interpolate between two camera pairs, meaning that you only need the bandwidth to deal with four camera views at any time. This applies no matter how many cameras you have in the ring (e.g., you could have a ring with many more cameras to get higher quality and still only need to have the GPU/disk/network bandwidth to handle four at a time).
3) The more cameras you have the better the stereo quality. In the limit of an infinite number of cameras (a camera pair for every possible viewing yaw), the stereo would be perfect. This is in contrast to stereo over-under panoramas, where the stereo is fundamentally incorrect and increasing the panorama resolution will not fix it even in the limit.
The main limitation is that it only correctly supports yawing head rotations. Pitching your head will usually look fine, but strictly speaking the representation does not properly take into account the forward offset of your eyes. Rolling your head will cause problems.
Right now I have stereo panorama stills working, but I've only done a quick-and-dirty camera calibration, so there are distracting misalignments and seams between the cameras.
Edit: removed demos while I improve the calibration
26 Replies
- This looks fantastic as it goes towards videos that're truly viewable in the Rift. What is the vertical FoV of the entire device? How far up can you look before it starts to clip out pieces that the cameras can't see?
- This looks awesome.
- This is amazing. Even more amazing, it's not nearly as expensive as it looks. Looking forward to seeing your videos!
- That's something I'm looking forward to! Looking great and I can't wait to test some of your videos.
- I was wondering if anyone was working on such a rig, nice work. I was going to attempt this myself, but plenty of people have been on this for awhile now.
For example,
http://makingview.no/makingview.com/?page_id=23 try their app out too, it works very well.
and
http://www.airpano.com/360Degree-VideoTour.php?3D=Video-Manhattan-New-York
combining this with the Rift is really exciting and I see a HUGE amount of potential here. I look forward to seeing how your videos turn out. - I'd like to help with this project as it can apply to remote controlling robotics.
- Any progress on this? Would love to see the result cause I'm thinking of making something similar as well
- So, bad news: the raspberry pi cameras have a terrible limitation that I didn't know about, which is that when taking video, they crop to the center of the sensor rather than sampling the full sensor image. Which means that, for a 1920x1080 crop out of a 2592x1944 sensor, it absolutely kills the field of view, even with fisheye lenses. This is a software issue that will allegedly be fixed eventually for the raspberry pi, but I'm not holding my breath.
But, the good news: I've almost completed another prototype, based on a different camera module, which can take video in full hd with a good field of view. As a side benefit, the new cameras are much nicer in general, especially in low-light conditions. However, they are a lot more expensive, at ~$160 each compared to $70 for the raspberry pi + camera combination (the new cameras do their own video encoding, so this also eliminates the need to have a ton of raspberry pis, which were irritatingly unreliable anyway).
So in the next couple of weeks expect to get some sample video. I'll probably port the viewer to webgl and vr.js just for convenience. - Hi Pyry,
I've been hoping someone that's researching or developing 360 stereoscopic 3D could answer this. Here's a quote of a reply I made on the panocam3d thread"nukemarine" wrote:
Starting this off with saying I'm not an engineer or programmer, but I'm interested not only in how the 3D 360 stitching is done but also in if it can be improved.
From the looks of the camera, it seems you're using 12 cameras. 6 pairs are positioned so each pair have the same offset and are parallel to each other. 6 cameras used for the right eye and 6 for the left eye. I assume then that the software stiching uses ~60 degrees of the center view from each camera to get the full 360.
What I'm wondering is could it be better to use all 12 cameras for each eye. Instead of parallel placement, they're evenly spread out in 12 directions. The software stitches ~30 degrees from each camera for the right eye and also for the left eye. However, to get the Stereo 3D, the software uses a different portion of the image from each camera base on the FOV of what the each eye would see if you had turned your head to move your eye to that camera spot.
So, if each camera covers 100 degrees, the computer would grab images from 10 to 40 for the right eye and 60 to 90 for the left eye to generate the 360 for each eye. Since more images are used, there's less visible distortion when turning. By using different portions of the picture, you create the visual offset needed for stereo 3D.
Of course, I'm just throwing out numbers without references or measurements. I would assume you'll want to use a model based on an average human head with an average IPD turning around in a circle. Just a rough drawing I did yesterday seems to show this is doable.
My questions then would be: Does what I wrote make sense? Does the 360 3D camera work in the first instance (6 pairs of parallel cameras) or sort of in the way I'm decribing with using all 12 cameras for each eye? If the option I describe works better, can the 360 3D camera do it as designed with slightly different stiching protocol or will the hardware need to be modified? Would decreasing the FOV for each camera since stitching ignores a large part of the image improve the overall stitched image? Since the lens would be using different FOV, could a modified lens work to improve the portion of the image used for both right and left eyes?
The questions at the end can still apply in your case as you're also designing such a camera. Is it mandatory to pair up the cameras or can different portions of the view be used to generate the 360 stereoscopic 3D? - You can point fisheye lenses out radially. From the overlapping view from any two cameras, dewarp and match a vertical strip for a stereoscopic view (left side of rightward camera view and right side of leftward camera view). They do not need to be "paired up", however the post processing is less intensive with paired cameras, and you get less warp distortion. A classic hardware versus software tradeoff (with some potential loss of image quality when using fewer cameras)...
In other words, you can use multiple cameras to simultaneously simulate a stereo panorama taken with a single camera:
http://www.researchgate.net/publication/2608330_Stereo_Panorama_with_a_Single_Camera/file/32bfe512645b3128eb.pdf
Of course, this way really requires lots more images to get a good quality stereo panorama shot. And you may need a lot of manually adjusted SIFT points to get the images all aligned correctly:
http://www.ijmer.com/papers/Vol2_Issue5/AO2532013206.pdf
