Forum Discussion
Keeping bearing constant without cropping with 3d movies
Short answer - track the movie and re-render it into a 360 panoramic format. Consider this movie (in cross-eyed side by side equirectangular format). https://www.youtube.com/watch?v=3JXi175-W...
How I did this: the camera is on a 2-axis brushless gimbal stabilizer which is stabilizing roll and pitch. Roll and pitch are constant (or almost) at zero approximately (ie. the camera is level). So when I convert the frames of the movies from 180 by 150 fisheye to 180 by 180 equirectangular the camera movement only consists of yaw jitter and global yaw movements (maybe a maximum of 120 degrees). The yaw movements, because the camera is level, correspond to linearly angular proportional (because of the equirectangular format) lateral movements of the frame. So you could stabilize the shot just in After Effects say by finding a distant point that was visible throughout the shot and using the Tracker in After Effects to keep it in a constant lateral position. Or just use Warp Stabilizer directly.
Here is a video where Ive done that.
https://www.youtube.com/watch?v=JgadcsQ9xEQ
You see here it has removed yaw jitter but not all global yaw movement.
Here it is with uncorrected yaw jitter
https://www.youtube.com/watch?v=ovIifgJb4uw
(there is a lot of jitter here because I am holding the camera and stabilizer over my head on a pole -- in the crowded park scene I am controlling yaw jitter better with a strap under tension around my neck attached to handles I am holding on either side of the stabilizer rig)
Of course mechanically I could use a 3-axis gimbal stabilizer configuration and remove yaw jitter entirely -- and yaw global movements too if I set the controller to fixed yaw, rather than Follow mode (where it smoothly follows your apparent intended yaw movements). But 3-axis is overkill for many uses -- it is heavier, more bulky, harder to transport safely, uses more batteries etc etc. And you cannot convert the 180 frames to a fixed bearing 360 experience as the yaw is constant with Fixed yaw. But for a 360 camera system Fixed yaw is obviously the way to go if you can work out the engineering of mechanically stabilizing it.
Ok, but how to handle the general case, where the camera is level but there are random yaw (global and jitter ) movements, and there are no distant features visible throughout, and you want to maintain constant bearing (forget about conversion to 360 and wrap enabling for the minute.) Well you could track the footage with a Camera Tracker. Like is in Blender 3d now.
http://wiki.blender.org/index.php/Doc:2.6/Manual/Motion_Tracking
but this needs rectilinear images to work with. So you have to convert your frames to rectilinear format, say 120 degrees by 120 degrees -- in PTGui or Hugin say. Which is time consuming. And then you have to track the footage and export the camera track in a format with humanly readable yaw values. Then you need to massage that into a script which will apply a lateral transform to the equirectangular images in After Effects (which last sentence is beyond my limited After Effects and script editing skills). You could use the Offset filter in After Effects to produce the wrap effect beyond 180 degree transforms for 360 images.
So that is a general solution, and it should work ok, as you have pretty stable footage already, you know the fov, there is relatively little rolling shutter, and the view is very wide angle so there will be plenty of detail to track, even after you have masked out busy foreground areas in the tracking say.
But what I did was use PTGui (or you could use Hugin) to do the whole thing quickly with the 180 by 180 equirectangular images I ended up with after batch processing the original aligned (L/R) fisheye images. I start with all the 180 by 180 equirectangular left images say in a Project. In PTGui (or Hugin I think) you can specify what parameters you want to track (as part of the alignment of images prior to the stitching phase in normal usage). You can track roll, pitch, or yaw, or horizontal or vertical offset, or distortion factors. Here I just want to track yaw so you uncheck all the other variables. And I specified the output panorama format as 360 by 180 equirectangular. In this park scene there are more or less distant trees visible throughout the sequence -- not the same trees, but from frame to frame there are distant trees. So I used the masking features in PTGui to constrain the process of finding common points between the successive frames to an area in a small top middle section of the frames, away from the foreground movements. Because the camera is weaving about quite a lot PTGui constantly finds a new lot of common features. Usually in panorama calibration and stitching the assumption by the program is that the camera movement is approximately parallax free -- which is not the case at all here. But by having a small point finding area with distant features this assumption is close enough here. Anyway, after you have found all the points you do the Optimization step for yaw only and output the 360/180 Panorama(s) as separate frames (not stitched) and that is the end of the job. It has removed yaw jitter and yaw global movements and converted the images to 360 by 180 and automatically done the wrap. Then you save that conversion as a template and apply it to the Right images.
So this will only work reliably and accurately if you have distant buildings, trees etc throughout the shot.
Here is a video where Ive done that.
https://www.youtube.com/watch?v=JgadcsQ9xEQ
You see here it has removed yaw jitter but not all global yaw movement.
Here it is with uncorrected yaw jitter
https://www.youtube.com/watch?v=ovIifgJb4uw
(there is a lot of jitter here because I am holding the camera and stabilizer over my head on a pole -- in the crowded park scene I am controlling yaw jitter better with a strap under tension around my neck attached to handles I am holding on either side of the stabilizer rig)
Of course mechanically I could use a 3-axis gimbal stabilizer configuration and remove yaw jitter entirely -- and yaw global movements too if I set the controller to fixed yaw, rather than Follow mode (where it smoothly follows your apparent intended yaw movements). But 3-axis is overkill for many uses -- it is heavier, more bulky, harder to transport safely, uses more batteries etc etc. And you cannot convert the 180 frames to a fixed bearing 360 experience as the yaw is constant with Fixed yaw. But for a 360 camera system Fixed yaw is obviously the way to go if you can work out the engineering of mechanically stabilizing it.
Ok, but how to handle the general case, where the camera is level but there are random yaw (global and jitter ) movements, and there are no distant features visible throughout, and you want to maintain constant bearing (forget about conversion to 360 and wrap enabling for the minute.) Well you could track the footage with a Camera Tracker. Like is in Blender 3d now.
http://wiki.blender.org/index.php/Doc:2.6/Manual/Motion_Tracking
but this needs rectilinear images to work with. So you have to convert your frames to rectilinear format, say 120 degrees by 120 degrees -- in PTGui or Hugin say. Which is time consuming. And then you have to track the footage and export the camera track in a format with humanly readable yaw values. Then you need to massage that into a script which will apply a lateral transform to the equirectangular images in After Effects (which last sentence is beyond my limited After Effects and script editing skills). You could use the Offset filter in After Effects to produce the wrap effect beyond 180 degree transforms for 360 images.
So that is a general solution, and it should work ok, as you have pretty stable footage already, you know the fov, there is relatively little rolling shutter, and the view is very wide angle so there will be plenty of detail to track, even after you have masked out busy foreground areas in the tracking say.
But what I did was use PTGui (or you could use Hugin) to do the whole thing quickly with the 180 by 180 equirectangular images I ended up with after batch processing the original aligned (L/R) fisheye images. I start with all the 180 by 180 equirectangular left images say in a Project. In PTGui (or Hugin I think) you can specify what parameters you want to track (as part of the alignment of images prior to the stitching phase in normal usage). You can track roll, pitch, or yaw, or horizontal or vertical offset, or distortion factors. Here I just want to track yaw so you uncheck all the other variables. And I specified the output panorama format as 360 by 180 equirectangular. In this park scene there are more or less distant trees visible throughout the sequence -- not the same trees, but from frame to frame there are distant trees. So I used the masking features in PTGui to constrain the process of finding common points between the successive frames to an area in a small top middle section of the frames, away from the foreground movements. Because the camera is weaving about quite a lot PTGui constantly finds a new lot of common features. Usually in panorama calibration and stitching the assumption by the program is that the camera movement is approximately parallax free -- which is not the case at all here. But by having a small point finding area with distant features this assumption is close enough here. Anyway, after you have found all the points you do the Optimization step for yaw only and output the 360/180 Panorama(s) as separate frames (not stitched) and that is the end of the job. It has removed yaw jitter and yaw global movements and converted the images to 360 by 180 and automatically done the wrap. Then you save that conversion as a template and apply it to the Right images.
So this will only work reliably and accurately if you have distant buildings, trees etc throughout the shot.