Showing results for 
Search instead for 
Did you mean: 

Stabilizing 2d and 3d panoramic videos

Level 4
I thought I would start a new topic on panoramic video stabilization -- hardware and software -- if for nothing else to keep all my links to this in one place.

Videostitch is a popular (but not cheap) program for stitching panoramic videos (2D) from camera arrays -- like from Gopro or Elmo QBIC rigs. There is a demo version (of the latest beta which includes stabilization) which will output a limited resolution video. It is pretty intuitive in operation if you are familiar with PTGui which is used beforehand to create stitching templates. As you can see by the initial graphic on the home page in the interface there is a tab where you can see tracking data for roll, pitch and yaw through the duration of the stitched shot and you can apply this data to stabilize it. It works but it is by no means as powerful as what you could do with a dedicated 3d or some tracking programs where you could, say, remove all roll and pitch motion, set them both to zero, and apply smoothing to the yaw track so you get a level, smoothly turning video output. But the ease of use is great and it is, like the rest of Videostitch operations, extremely fast. However this is a beta release and it is not very stable and there is some discontent with modifications of the operation of the stabilizer in the latest beta and I got a thin black seam at the end of my 360 output in my testing. It is a big achievement though to have direct tracking of panoramic videos -- with a standard 3d program you would need to convert your equirectangular to an extracted rectilinear view and the tracking might not be as robust as Videostitch can achieve with the data from the whole 360 frame. I am not sure if you can input an already stitched panorama (like one of a pair of stereo 360 equirectangulars say) but I think you can. I think it can only handle 360 videos (unless you pad them out to 360).

PTGui (and Hugin) can be used to stabilize all sorts of (fixed position where you are trying to hold the camera steady in the same direction) panoramic videos if you convert the video first to frame sequences. On a pole in windy conditions for instance. I am only experienced with PTGui and I use it quite a lot for panoramic stabilization of this kind of shot. Here is a tutorial for stabilization with Hugin (free) but the rest of this section just applies to my experience with PTGui:
With PTGui you can level as well as stabilize the entire shot only. Not smooth out motion tracks. At least not so far as I have worked out. It is buggy with long sequences (say more than 600 large frames) on my hardware. It makes a difference to how long a sequence you can process if you create a proxy version of the frames at lower resolution (say 800 by 400 pixels) and then apply that as a template -- but the template can crash too. It is fast until it runs out of memory and then it becomes very slow. It helps if you first create control points between say the first 10 frames and the last 10 frames (by selecting those 20 frames) and then generate control points for the rest of the frames. This forces it to make common points between the last and first frames which otherwise it often might neglect to do. When you have optimised the sequence you output as separate frames.

Blender3D has a plugin (addon) which can be used for tracking and stabilizing panoramic (equirectangular 360) videos
I havent tried it but it looks very powerful. It uses the Camera Tracker in recent Blender versions to track 2 points which must remain visible throughout the sequence. These points should be separated quite a bit in the frame. It would be good for aerial stitched panoramas for instance I imagine. It doesnt level the panorama but produces a fixed roll, pitch and yaw throughout the video. You could level it later with PTGui, Videostitch etc (or with the built in Blender Cycles renderer maybe). I have an idea that the Cycles renderer is kind of slow for equirectangular output.

After Effects I use After Effects a lot for panoramic video stabilization. There are a variety of tools and methods you can use with it depending on what your source material is. If you are shooting handheld 180 fisheye stereo videos from a fixed location for instance and trying to hold the camera rig steady and pointing in the same direction you can do a much quicker stabilization job than you can with PTGui for instance (with long sequences).

With 180 fisheye stereo the workflow would be something like this. Crop the videos to a square, center and align the fisheye circles approximately. Apply Warp Stabilizer to one Layer (say the L video) . Use a mask and Precompose to restrict tracking to a non-motion filled area of the frame. Set the application of the Warp Stabilizer analysis to no motion, no scale, Perspective mode and non-crop, non-interpolate borders. Copy the motion track to the other Layer. This is good, compared with PTGui too, for these static shots, because it reduces rolling shutter. Then you can output as frame sequences and do the rest of the alignment and levelling and remapping to equirectangular say in PTGui.

With 180 "travelling" stereo videos -- where I am walking down a busy the street say -- with a 2 axis (roll and pitch) gimbal stabilized rig -- I sometimes use a different work flow. First I convert the video to a frame sequence and extract downward (or sometimes upward) looking wide angle rectilinear views (say 800 by 800 pixel) 120 degree views. Then I use the Camera Tracker in After Effects to solve for that sequence -- which it usually does a robust job with because the downward view is not so obscured with internal frame motion and has a lot of common features from frame to frame. Then I use an Expression to take the roll track data of this extracted rectilinear sequence to laterally shift (Transform x) an equirectangular converted version of the fisheye frames. This is much quicker than rendering it in a 3d program. If you are dealing with 360 equirectangular frames you can use the same kind of expression via an Offset filter so you dont crop the equirectangular view. This approach works because the video is basically level because of the gimbal stabilizer and you just need to remove yaw jitter and smooth global yaw movements. You can use the same kind of approach with an approximately level 360 2D video (overhead on a counterweighted pole say).

Autopano Video Pro AVP, made by Kolor, which I havent tried, but there is a demo version, has a stabilization feature which is described here:
You need Autopano, Kolor's panorama stitching progam too I think (according to this -- ). AVP's stabilizer has a controllable damping action so low levels remove jitter and higher levels reduce global movements. AVP (the Pro version) is GPU accelerated so it is fast with a good graphics card.

Level 4
Since my initial post I have done some testing with the Blender addon Panorama-Tracker -- which can be used for stabilization of stitched 360s and individual 180 fisheye videos (2D or 3D).

Here is a 360 (2D) video panorama I made with a two camera back to back Gopro + 185 fisheye rig -- with a short monopod held over my head.
You can see by the way the equirectangular video is waving about there is some camera motion and this would be disturbing in a HMD or in an interactive version on a screen. Here is the stabilized version in interactive form:
Here the buildings are rock steady. So Panorama-Tracker works well for scenes where there is a pair of points visible throughout the shot and the camera is basically at the same point in space.

What if the camera rig is moving through the scene? Here the use cases are more restricted. How Panorama-Tracker works is you add two points to the initial frame and track them through the frame sequence. Then you nominate one point as the Focus and one point as the Target. Then, after tracking them you can click a button and generate a Panoramic Camera which will incorporate the stabilized equirectangular sequence as an Environment Map. The add-on will keep the Focus point at a fixed location in the equirectangular frame and the angle from the Focus to the Target point will remain constant in the equirectangular output frame.

Say you have a situation where you are walking on a flat road up towards a tall building. And you choose as Focus point some feature 10m off the ground. As you get closer this will remain in the same location in the equirectangular frame so this will make the viewpoint tilt progressively in the processed video. You will have removed jitter but there will be a global movement you dont want.

There is one situation I can think of -- on level ground - where it will still work in a travelling video situation (in addition to cases where is there is very distant scenery). Suppose you can find a Focus point (which remains visible through the shot) which is at the same level as the initial viewpoint. And you choose a Target point which is vertically above or horizontal with the Focus point. Then if your initial equirectangular frame is level it will remain level throughout the sequence and be stabilized direction-wise as well (if you are heading towards the Focus point). I will test this soon.

Level 4
After Effects -- continued
It is a very good thing sometimes to be able to accurately track a panoramic video.

Then you can use it as a background in a VR application, for instance, and know that you can have CG objects and characters inserted into the scene with no giveaway differential motion between the background and the CG inserts. Also it permits accurate stabilization.

After Effects has a good camera tracker and it also can use equirectangular 360s as environment maps (with the Environment Map effect or as an Environment Map layer (in Raytracing 3d mode). If you import the equirectangular stitched 360 movie as a layer you can add the Environment Map effect, add a camera and simply and quickly render out rectilinear movies from a panoramic movie. After Effects cannot track equirectangular movies directly but you can use the Environment Map filter to export a forward facing rectilinear wide angle movie for tracking purposes. 120 degree square rectilinear about 800 by 800 pixels is good.

Then you can use the Camera Tracker to track that. Then in the Camera Tracker panel go Create Camera. Then you can delete the rectilinear tracking layer. Now change the Composition dimension to something a bit larger. Say 1000 by 1000 pixels.

If you want to stabilize the footage in After Effects the only way I have worked out how to do it simply is to use the Element3d plugin (third party). First import the original equirectangular 360 frame sequence. Make a solid layer the dimensions of the composition. Apply the Element3d filter to the Solid layer. Open the filter properties. Under Rendering choose the panorama equi layer as a source and check show Background. Open the Background orientation controllers in the Element3d filter. Whippick (an expressions tool) the Background orientation items in the Element 3d filter to the corresponding orientation items of the solved Tracker Camera (in the most obvious order).

Now the Tracker Camera orientation moves are mirrored by the Element 3d Background panoramic video sphere.

Now you need a new Camera to see how well your stabilization works. If all is well now you can render out the stabilized equirectangular panorama. You have to do this in cubic mode. ie. render out front, right, left, back, up , down 90 degree (stabilized) square image sequences with the new camera. Save each of these projects so you can load all 6 projects into a new project and batch the rendering.

Now you have to stitch these 6 stabilized sequences, the sides of the cube, into stabilized equirectangular 360s with PTGui. So use a file renamer to have the front images named: l_0001a, l_00002a etc, right images named: l_0001b, l_000b2 etc, left images named: l_001c etc ... then make a project in PTGui with the first set of six images of your cubes viz. l_0001a.jpg, l_0001b.jpg, l_0001c.jpg, l_0001d.jpg, l_0001e.jpg, l_0001f.jpg as the source images - and add the image settings in PTGui to make them stitch into a cube (90 degree yaws and tilts). Then put all the images into the front folders so they form 6 image sequences in alphabetical order. Now you can use the Batch Builder and Batch Stitcher tools in PTGui to output your equirectangulars. (It might be necessary when you generate the cubic sides to make them slightly oversize, like 95 degrees, otherwise you might have problems with visible seams.

Level 2
Very well done.
What an extremely complicated process.
VideoStitch can do a pretty good job at stabilizing. Does it help or hinder stabilizing in VideoStitch before figuring out camera tracks? If the purpose is to add 3D elements to the movie.
Video editing software needs to understand equirectangular format natively.

I think what is missing from a hardware stabilized video gimbo is a shock absorber. A pole inside a pole connected with a spring calibrated to the weight of the camera. Add a small motor to lift and lower the camera and dampen the movement.

Level 2
I can't find the "new Topic" button, but this seems a good place to post this.

The SteadXP Kickstarter opened for pledges today. This device records the attitude and acceleration of you camera while you shoot. Then, included software allows you to stabilize more-or-less automatically in post. You'll need to shoot a bit wider than you otherwise would, as there will be some cropping that will "zoom-in" a bit.

There are two models - one for GoPro Hero 2, 3, 3+, and 4, and another that will work with any cameras that output composite video while shooting, and that have a microphone input.

It seems to me that this could be valuable while shooting footage intended for VR display.

I have no affiliation with SteadXP, except that I pledged to buy one of each type.


Level 4
The SteadXP Kickstarter opened for pledges today. This device records the attitude and acceleration of you camera while you shoot. Then, included software allows you to stabilize more-or-less automatically in post. You'll need to shoot a bit wider than you otherwise would, as there will be some cropping that will "zoom-in" a bit.

Their sample videos are very impressive .. I am sure this will be a big deal for vr videos if they allow programs like After Effects and Nuke ingest the camera track data they produce. If they confine you to using their software or some expensive partner specialist video panorama software it will be an opportunity lost by them.