Stabilizing 2d and 3d panoramic videos
I thought I would start a new topic on panoramic video stabilization -- hardware and software -- if for nothing else to keep all my links to this in one place.
Videostitch http://www.video-stitch.com/ is a popular (but not cheap) program for stitching panoramic videos (2D) from camera arrays -- like from Gopro or Elmo QBIC rigs. There is a demo version (of the latest beta which includes stabilization) which will output a limited resolution video. It is pretty intuitive in operation if you are familiar with PTGui which is used beforehand to create stitching templates. As you can see by the initial graphic on the home page in the interface there is a tab where you can see tracking data for roll, pitch and yaw through the duration of the stitched shot and you can apply this data to stabilize it. It works but it is by no means as powerful as what you could do with a dedicated 3d or some tracking programs where you could, say, remove all roll and pitch motion, set them both to zero, and apply smoothing to the yaw track so you get a level, smoothly turning video output. But the ease of use is great and it is, like the rest of Videostitch operations, extremely fast. However this is a beta release and it is not very stable and there is some discontent with modifications of the operation of the stabilizer in the latest beta and I got a thin black seam at the end of my 360 output in my testing. It is a big achievement though to have direct tracking of panoramic videos -- with a standard 3d program you would need to convert your equirectangular to an extracted rectilinear view and the tracking might not be as robust as Videostitch can achieve with the data from the whole 360 frame. I am not sure if you can input an already stitched panorama (like one of a pair of stereo 360 equirectangulars say) but I think you can. I think it can only handle 360 videos (unless you pad them out to 360).
http://www.video-stitch.com/forum/index.php?page=Thread&threadID=245
PTGui (and Hugin) can be used to stabilize all sorts of (fixed position where you are trying to hold the camera steady in the same direction) panoramic videos if you convert the video first to frame sequences. On a pole in windy conditions for instance. I am only experienced with PTGui and I use it quite a lot for panoramic stabilization of this kind of shot. Here is a tutorial for stabilization with Hugin (free) but the rest of this section just applies to my experience with PTGui:
http://imgur.com/a/3qfWQ
With PTGui you can level as well as stabilize the entire shot only. Not smooth out motion tracks. At least not so far as I have worked out. It is buggy with long sequences (say more than 600 large frames) on my hardware. It makes a difference to how long a sequence you can process if you create a proxy version of the frames at lower resolution (say 800 by 400 pixels) and then apply that as a template -- but the template can crash too. It is fast until it runs out of memory and then it becomes very slow. It helps if you first create control points between say the first 10 frames and the last 10 frames (by selecting those 20 frames) and then generate control points for the rest of the frames. This forces it to make common points between the last and first frames which otherwise it often might neglect to do. When you have optimised the sequence you output as separate frames.
Blender3D has a plugin (addon) which can be used for tracking and stabilizing panoramic (equirectangular 360) videos
https://github.com/dfelinto/Panorama-Tracker
http://www.dalaifelinto.com/?p=966
http://www.autopano.net/wiki-en/action/view/Autopano_Video_-_Advanced_processing_with_Blender_and_After_Effects
I havent tried it but it looks very powerful. It uses the Camera Tracker in recent Blender versions to track 2 points which must remain visible throughout the sequence. These points should be separated quite a bit in the frame. It would be good for aerial stitched panoramas for instance I imagine. It doesnt level the panorama but produces a fixed roll, pitch and yaw throughout the video. You could level it later with PTGui, Videostitch etc (or with the built in Blender Cycles renderer maybe). I have an idea that the Cycles renderer is kind of slow for equirectangular output.
After Effects I use After Effects a lot for panoramic video stabilization. There are a variety of tools and methods you can use with it depending on what your source material is. If you are shooting handheld 180 fisheye stereo videos from a fixed location for instance and trying to hold the camera rig steady and pointing in the same direction you can do a much quicker stabilization job than you can with PTGui for instance (with long sequences).
With 180 fisheye stereo the workflow would be something like this. Crop the videos to a square, center and align the fisheye circles approximately. Apply Warp Stabilizer to one Layer (say the L video) . Use a mask and Precompose to restrict tracking to a non-motion filled area of the frame. Set the application of the Warp Stabilizer analysis to no motion, no scale, Perspective mode and non-crop, non-interpolate borders. Copy the motion track to the other Layer. This is good, compared with PTGui too, for these static shots, because it reduces rolling shutter. Then you can output as frame sequences and do the rest of the alignment and levelling and remapping to equirectangular say in PTGui.
With 180 "travelling" stereo videos -- where I am walking down a busy the street say -- with a 2 axis (roll and pitch) gimbal stabilized rig -- I sometimes use a different work flow. First I convert the video to a frame sequence and extract downward (or sometimes upward) looking wide angle rectilinear views (say 800 by 800 pixel) 120 degree views. Then I use the Camera Tracker in After Effects to solve for that sequence -- which it usually does a robust job with because the downward view is not so obscured with internal frame motion and has a lot of common features from frame to frame. Then I use an Expression to take the roll track data of this extracted rectilinear sequence to laterally shift (Transform x) an equirectangular converted version of the fisheye frames. This is much quicker than rendering it in a 3d program. If you are dealing with 360 equirectangular frames you can use the same kind of expression via an Offset filter so you dont crop the equirectangular view. This approach works because the video is basically level because of the gimbal stabilizer and you just need to remove yaw jitter and smooth global yaw movements. You can use the same kind of approach with an approximately level 360 2D video (overhead on a counterweighted pole say).
Autopano Video Pro AVP, made by Kolor, which I havent tried, but there is a demo version, has a stabilization feature which is described here:
http://www.autopano.net/wiki-en/action/view/Autopano_Video_-_Anti_shak
You need Autopano, Kolor's panorama stitching progam too I think (according to this -- http://freedom360.us/shop/kolor-autopano-autopano-video/ ). AVP's stabilizer has a controllable damping action so low levels remove jitter and higher levels reduce global movements. AVP (the Pro version) is GPU accelerated so it is fast with a good graphics card.
Videostitch http://www.video-stitch.com/ is a popular (but not cheap) program for stitching panoramic videos (2D) from camera arrays -- like from Gopro or Elmo QBIC rigs. There is a demo version (of the latest beta which includes stabilization) which will output a limited resolution video. It is pretty intuitive in operation if you are familiar with PTGui which is used beforehand to create stitching templates. As you can see by the initial graphic on the home page in the interface there is a tab where you can see tracking data for roll, pitch and yaw through the duration of the stitched shot and you can apply this data to stabilize it. It works but it is by no means as powerful as what you could do with a dedicated 3d or some tracking programs where you could, say, remove all roll and pitch motion, set them both to zero, and apply smoothing to the yaw track so you get a level, smoothly turning video output. But the ease of use is great and it is, like the rest of Videostitch operations, extremely fast. However this is a beta release and it is not very stable and there is some discontent with modifications of the operation of the stabilizer in the latest beta and I got a thin black seam at the end of my 360 output in my testing. It is a big achievement though to have direct tracking of panoramic videos -- with a standard 3d program you would need to convert your equirectangular to an extracted rectilinear view and the tracking might not be as robust as Videostitch can achieve with the data from the whole 360 frame. I am not sure if you can input an already stitched panorama (like one of a pair of stereo 360 equirectangulars say) but I think you can. I think it can only handle 360 videos (unless you pad them out to 360).
http://www.video-stitch.com/forum/index.php?page=Thread&threadID=245
PTGui (and Hugin) can be used to stabilize all sorts of (fixed position where you are trying to hold the camera steady in the same direction) panoramic videos if you convert the video first to frame sequences. On a pole in windy conditions for instance. I am only experienced with PTGui and I use it quite a lot for panoramic stabilization of this kind of shot. Here is a tutorial for stabilization with Hugin (free) but the rest of this section just applies to my experience with PTGui:
http://imgur.com/a/3qfWQ
With PTGui you can level as well as stabilize the entire shot only. Not smooth out motion tracks. At least not so far as I have worked out. It is buggy with long sequences (say more than 600 large frames) on my hardware. It makes a difference to how long a sequence you can process if you create a proxy version of the frames at lower resolution (say 800 by 400 pixels) and then apply that as a template -- but the template can crash too. It is fast until it runs out of memory and then it becomes very slow. It helps if you first create control points between say the first 10 frames and the last 10 frames (by selecting those 20 frames) and then generate control points for the rest of the frames. This forces it to make common points between the last and first frames which otherwise it often might neglect to do. When you have optimised the sequence you output as separate frames.
Blender3D has a plugin (addon) which can be used for tracking and stabilizing panoramic (equirectangular 360) videos
https://github.com/dfelinto/Panorama-Tracker
http://www.dalaifelinto.com/?p=966
http://www.autopano.net/wiki-en/action/view/Autopano_Video_-_Advanced_processing_with_Blender_and_After_Effects
I havent tried it but it looks very powerful. It uses the Camera Tracker in recent Blender versions to track 2 points which must remain visible throughout the sequence. These points should be separated quite a bit in the frame. It would be good for aerial stitched panoramas for instance I imagine. It doesnt level the panorama but produces a fixed roll, pitch and yaw throughout the video. You could level it later with PTGui, Videostitch etc (or with the built in Blender Cycles renderer maybe). I have an idea that the Cycles renderer is kind of slow for equirectangular output.
After Effects I use After Effects a lot for panoramic video stabilization. There are a variety of tools and methods you can use with it depending on what your source material is. If you are shooting handheld 180 fisheye stereo videos from a fixed location for instance and trying to hold the camera rig steady and pointing in the same direction you can do a much quicker stabilization job than you can with PTGui for instance (with long sequences).
With 180 fisheye stereo the workflow would be something like this. Crop the videos to a square, center and align the fisheye circles approximately. Apply Warp Stabilizer to one Layer (say the L video) . Use a mask and Precompose to restrict tracking to a non-motion filled area of the frame. Set the application of the Warp Stabilizer analysis to no motion, no scale, Perspective mode and non-crop, non-interpolate borders. Copy the motion track to the other Layer. This is good, compared with PTGui too, for these static shots, because it reduces rolling shutter. Then you can output as frame sequences and do the rest of the alignment and levelling and remapping to equirectangular say in PTGui.
With 180 "travelling" stereo videos -- where I am walking down a busy the street say -- with a 2 axis (roll and pitch) gimbal stabilized rig -- I sometimes use a different work flow. First I convert the video to a frame sequence and extract downward (or sometimes upward) looking wide angle rectilinear views (say 800 by 800 pixel) 120 degree views. Then I use the Camera Tracker in After Effects to solve for that sequence -- which it usually does a robust job with because the downward view is not so obscured with internal frame motion and has a lot of common features from frame to frame. Then I use an Expression to take the roll track data of this extracted rectilinear sequence to laterally shift (Transform x) an equirectangular converted version of the fisheye frames. This is much quicker than rendering it in a 3d program. If you are dealing with 360 equirectangular frames you can use the same kind of expression via an Offset filter so you dont crop the equirectangular view. This approach works because the video is basically level because of the gimbal stabilizer and you just need to remove yaw jitter and smooth global yaw movements. You can use the same kind of approach with an approximately level 360 2D video (overhead on a counterweighted pole say).
Autopano Video Pro AVP, made by Kolor, which I havent tried, but there is a demo version, has a stabilization feature which is described here:
http://www.autopano.net/wiki-en/action/view/Autopano_Video_-_Anti_shak
You need Autopano, Kolor's panorama stitching progam too I think (according to this -- http://freedom360.us/shop/kolor-autopano-autopano-video/ ). AVP's stabilizer has a controllable damping action so low levels remove jitter and higher levels reduce global movements. AVP (the Pro version) is GPU accelerated so it is fast with a good graphics card.