Keeping bearing constant without cropping with 3d movies

That's great! It works very well! This is a great way to be in someone's feet and I feel it has a lot of narrative potential for movies and "empathic experiences"!
What's your workflow for this? Which software do you use for tracking the yaw? And for converting to 360?
Thanks for making all your work public!

That makes a surprising amount of difference in the experience. I would also be interested in more detail about the process.

This is pretty interesting. I'm very impressed with the idea, and how well this particular movie was converted. It could be a challenge to help the user know where the action is, as when I was watching the converted movie, I kept being surprised by the edge of the video sweeping into my view -- "oh! things are moving in a different direction now".

If we're not worried about breaks in immersion due to the nature of this not filling the full 360x180, maybe a virtual compass could be there in front of the user and it will show up with the "correct direction" when the user isn't facing it.

Maybe render a virtual nose?

How I did this: the camera is on a 2-axis brushless gimbal stabilizer which is stabilizing roll and pitch. Roll and pitch are constant (or almost) at zero approximately (ie. the camera is level). So when I convert the frames of the movies from 180 by 150 fisheye to 180 by 180 equirectangular the camera movement only consists of yaw jitter and global yaw movements (maybe a maximum of 120 degrees). The yaw movements, because the camera is level, correspond to linearly angular proportional (because of the equirectangular format) lateral movements of the frame. So you could stabilize the shot just in After Effects say by finding a distant point that was visible throughout the shot and using the Tracker in After Effects to keep it in a constant lateral position. Or just use Warp Stabilizer directly.
Here is a video where Ive done that.
https://www.youtube.com/watch?v=JgadcsQ9xEQ

You see here it has removed yaw jitter but not all global yaw movement.

Here it is with uncorrected yaw jitter
https://www.youtube.com/watch?v=ovIifgJb4uw
(there is a lot of jitter here because I am holding the camera and stabilizer over my head on a pole -- in the crowded park scene I am controlling yaw jitter better with a strap under tension around my neck attached to handles I am holding on either side of the stabilizer rig)

Of course mechanically I could use a 3-axis gimbal stabilizer configuration and remove yaw jitter entirely -- and yaw global movements too if I set the controller to fixed yaw, rather than Follow mode (where it smoothly follows your apparent intended yaw movements). But 3-axis is overkill for many uses -- it is heavier, more bulky, harder to transport safely, uses more batteries etc etc. And you cannot convert the 180 frames to a fixed bearing 360 experience as the yaw is constant with Fixed yaw. But for a 360 camera system Fixed yaw is obviously the way to go if you can work out the engineering of mechanically stabilizing it.

Ok, but how to handle the general case, where the camera is level but there are random yaw (global and jitter ) movements, and there are no distant features visible throughout, and you want to maintain constant bearing (forget about conversion to 360 and wrap enabling for the minute.) Well you could track the footage with a Camera Tracker. Like is in Blender 3d now.
http://wiki.blender.org/index.php/Doc:2.6/Manual/Motion_Tracking
but this needs rectilinear images to work with. So you have to convert your frames to rectilinear format, say 120 degrees by 120 degrees -- in PTGui or Hugin say. Which is time consuming. And then you have to track the footage and export the camera track in a format with humanly readable yaw values. Then you need to massage that into a script which will apply a lateral transform to the equirectangular images in After Effects (which last sentence is beyond my limited After Effects and script editing skills). You could use the Offset filter in After Effects to produce the wrap effect beyond 180 degree transforms for 360 images.

So that is a general solution, and it should work ok, as you have pretty stable footage already, you know the fov, there is relatively little rolling shutter, and the view is very wide angle so there will be plenty of detail to track, even after you have masked out busy foreground areas in the tracking say.

But what I did was use PTGui (or you could use Hugin) to do the whole thing quickly with the 180 by 180 equirectangular images I ended up with after batch processing the original aligned (L/R) fisheye images. I start with all the 180 by 180 equirectangular left images say in a Project. In PTGui (or Hugin I think) you can specify what parameters you want to track (as part of the alignment of images prior to the stitching phase in normal usage). You can track roll, pitch, or yaw, or horizontal or vertical offset, or distortion factors. Here I just want to track yaw so you uncheck all the other variables. And I specified the output panorama format as 360 by 180 equirectangular. In this park scene there are more or less distant trees visible throughout the sequence -- not the same trees, but from frame to frame there are distant trees. So I used the masking features in PTGui to constrain the process of finding common points between the successive frames to an area in a small top middle section of the frames, away from the foreground movements. Because the camera is weaving about quite a lot PTGui constantly finds a new lot of common features. Usually in panorama calibration and stitching the assumption by the program is that the camera movement is approximately parallax free -- which is not the case at all here. But by having a small point finding area with distant features this assumption is close enough here. Anyway, after you have found all the points you do the Optimization step for yaw only and output the 360/180 Panorama(s) as separate frames (not stitched) and that is the end of the job. It has removed yaw jitter and yaw global movements and converted the images to 360 by 180 and automatically done the wrap. Then you save that conversion as a template and apply it to the Right images.

So this will only work reliably and accurately if you have distant buildings, trees etc throughout the shot.

This is very, very interesting. However, does that mean that every shot you have you need to have something like building or trees in the distance. That feels a bit limiting to the content you are able to produce. Would following the more time consuming method you outlined above be more flexible?

Yes using a proper 3d tracker rather than a panorama stitcher program is the thing to do for universal situations.

I think the 3d Camera Tracker in After Effects itself will do a good job after some testing. If you convert your aligned equirectangular 180/180s to rectilinear 120/120 degrees AE made a very accurate looking 3d track (x, y, z, roll, pitch and yaw, fov, plus scene feature locations) -- that is, after masking out foreground motion. So the problem is to then how to render out the constant bearing 360/180 equirectangulars from After Effects --

[edit] which I have worked out eventually after learning a little bit about After Effects Expressions scripting -- what you need to do is take the value of the yaw data in degrees from the 3d Camera Tracker and use that to shift the 180/180 equirectangular frames (in 2D mode) left or right the correct number of pixels in a 360/180 composition window. For instance my script for the Position Transform values of the layer containing the 180/180 frames is

newh = thisComp.layer("3D Tracker Camera").orientation[1] - 50.5;
newv=position[1];
[1800 + 10*newh, newv]

and here is the result
http://www.youtube.com/watch?v=AcaXdI1bdLI

this is a very fast step once you have got the 3D Camera Track, and you are not rerendering the panorama in 3d so the quality is not affected at all.

Put your mouse cursor over the large skyscraper to the right of the movie. You can see there is no yaw jitter and minor global yaw drift.



BTW. ..... Also I notice there is this very cool freeware PC app for wireless modem remote control of Alexmos gimbal controller boards. Very sophisticated camera moves could be programmed with this that could be converted without software tracking to constant bearing 360 movies
https://www.youtube.com/watch?v=DmkmtFvnwCM#t=16

BTW also ... the latest Blender 3d has a different distortion model available in the Camera Tracker which is much better suited to fisheye lenses with strong distortion they say
http://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.71/More_Features
(under Motion Tracking)

Also when I was looking at tracking workflows I tried the After Effects to Blender script here and it worked ok with the latest Blender .. at least all the camera track type stuff looked like it was exported properly when I opened it in Blender
http://blenderartists.org/forum/showthread.php?249084-After-Effects-camera-into-Blender

Also I have put another couple of example conversion videos and downloads on the Oculus sub-Reddit
http://www.reddit.com/r/oculus/comments/28ifft/tracked_rift_3d180_panoramic_movies_you_can/

Also I have been trying Matchmover from Autodesk which is free now. It did an ok job too. Seems about as fast/slow as the 3d Tracker in After Effects. For like 3000 frames it takes about an hour for track and solve.