Skip to main content
Pyry
Up and Coming
September 25, 2013

Towards practical super-wide (180+) stereo video

Update:
Videos (you will need to have VR.js installed, preferable on Chrome, since the performance on Firefox is not as good):
Fondue party
Porch
Lab & crystal cove prototype
Robot soccer

I've been working on a stereo video panorama camera. It's essentially a ring camera array, with the cameras arranged as stereo pairs on the faces of a hexagon.

panocam_front_small.jpg

To do a full 360 degrees would take twelve cameras. Currently I'm only working with six cameras (three pairs), which gives about 240 degrees total horizontal FOV (120 degrees of head rotation + 120 degrees FOV in any particular viewpoint). Each camera is controlled by a Raspberry Pi, and then all the Pis are networked together-- this seems to be by far the most economical camera solution at only $70 per camera+pi (or $420 for all six). Since the Raspberry Pi cameras have a very low FOV, I've put a fisheye lens on each one (iPhone fisheye lenses, actually [~$6 each]), which bumps the horizontal FOV of each camera up to ~120.

panocam_overhead_small.jpg

Arranging the cameras in this kind of ring has a number of benefits:
1) The cameras can't see each other/can't occlude one another.
2) For any viewing direction (yaw) you only have to interpolate between two camera pairs, meaning that you only need the bandwidth to deal with four camera views at any time. This applies no matter how many cameras you have in the ring (e.g., you could have a ring with many more cameras to get higher quality and still only need to have the GPU/disk/network bandwidth to handle four at a time).
3) The more cameras you have the better the stereo quality. In the limit of an infinite number of cameras (a camera pair for every possible viewing yaw), the stereo would be perfect. This is in contrast to stereo over-under panoramas, where the stereo is fundamentally incorrect and increasing the panorama resolution will not fix it even in the limit.

The main limitation is that it only correctly supports yawing head rotations. Pitching your head will usually look fine, but strictly speaking the representation does not properly take into account the forward offset of your eyes. Rolling your head will cause problems.

Right now I have stereo panorama stills working, but I've only done a quick-and-dirty camera calibration, so there are distracting misalignments and seams between the cameras.

Edit: removed demos while I improve the calibration

    26 replies

    Show previous replies
    mediavr
    Up and Coming
    May 22, 2014
    "StephaneA" wrote:
    It seems that Jaunt doesn't shoot in stereo but rather rebuild the depth in post production from the camera array.
    Does any body knows anything about depth extraction from multiple cameras ?
    http://www.jauntvr.com/technology


    I dont see where they are saying that but if they are ... then they might use programs like
    http://www.stereoscopynews.com/hotnews/3d-cinema/post-production/3452-yuvsoft-stereo-processing-suite.html

    or
    http://www.stereoscopynews.com/hotnews/3d-technology/2d-to-3d-conversion/3074-fine-tune-your-depth-maps-with-dimenco-depth-2-0.html

    or
    http://www.stereoscopynews.com/hotnews/3d-technology/software-a-hardware-tools/2695-triaxes-stereotracer-converts-stereo-to-multiview.html

    or Kolor
    http://www.kolor.com/blog-en/2012/09/07/kolor-labs-measurement-using-panoramas-by-alexandre-jenny/

    or Agisoft's Photoscan 3d (Pro version) which supports video photogrammetry
    j1vvy
    Up and Coming
    May 23, 2014
    Jaunt VR describes the process as I expected and what I hope to do.
    Capturing every pixel at least three time and generating depthmap from this data. He is calling it computational photography.
    http://www.youtube.com/watch?v=X_Zf0LSNFGs&feature=share&t=6m55s
    muglore
    Up and Coming
    May 25, 2014
    depth map triangulation seems like a promising development that's getting more traction. I was thinking that time of flight cameras might be the way to go, but this software-based method of creating depth maps seems much more achievable for all of us.

    It would be nice if one of these companies would release some depth mapping software for us to play with!
    Golf Masters DK2 Alpha, GOLF MASTERS
    Fredz
    Up and Coming
    May 25, 2014
    "StephaneA" wrote:
    Does any body knows anything about depth extraction from multiple cameras ?

    Have a look here for state of the art depth extraction algorithms from stereo images : http://vision.middlebury.edu/stereo/eval/
    mediavr
    Up and Coming
    May 31, 2014
    Here is a free tool for semi-automatic depth map extraction from stereo pairs
    http://3dstereophoto.blogspot.com.au/

    ... the latest version is meant to be a lot faster than before.

    With 2D to 3D conversion for movies constructing high quality key frame depth maps and then propagating the depth for the intermediate frames is a common workflow ... so it is good tools are emerging to do the key frames more accurately.
    Lindblum
    Up and Coming
    June 19, 2014
    I should try out some of the depth-mapping tools mentioned here. I am currently playing around with AVISynth scripting. Here someone wrote a script that repurposes a motion-detection plugin for depth-mapping. Here is my more generalized version:

    LoadPlugin("mvtools2.dll")

    V = DirectShowSource("YOUR_SBS_VIDEO_FILE")
    #V = LanczosResize(V, Int(Width(V)/2),Int(Height(V)/2))	#Reduce size to reduce analysis computation
    W = Width(V)
    W2 = Int(W/2)
    H = Height(V)
    R = Crop(V, W2,0, 0,0)
    L = Crop(V, 0,0, W2,0)
    S = Interleave(L,R).MSuper()
    A = MAnalyse(S, isb=false, blksize=4)
    D = MMask(S, A, 100, 1.0, 3).SelectEven().Invert().GreyScale().Crop(0, 0, -16, H).LanczosResize(W2,H)
    StackHorizontal(L,D)

    Of course the results can get fuzzy, maybe more scripting can clean it up, but it works decently in real-time, even in VR Player.

    Terms & ConditionsAccessibility statement