First some background: I love the rift. In doing my own due diligence to figure out how the next form of vr would take hold in the marketplace and doing my own prototyping until the rft came along and I knew this was where it was at in terms of bleeding edge. However I am a filmmaker first. I have been a gamer my whole life and immediately understood the implications of why it needs to take root in that market first but in terms of what I need to do it's more about the storytelling in a forced perspective of filmmakeing rather than the free form of game worlds.That said I am of a pretty difficult crossroads of how to program for this medium.My first instinct was to just lock down 2 hd cameras and have the viewer join the story in a forced 3d perspective but part of me feels that would take away the freedom of the head tracking. So I went on a quest to see what could be done. As such I'm at a crossroads.I have seen a few different versions of 360 cameras. I think the data that is captured by these devices could be mapped in such a way that the audience could have the ability to look where they want but something might be lost in the 3d aspect as opposed to a 2 camera set up with a forced perspective.My other idea was to record 2 3d cameras side by side to allow full head tracking but t might cause some visual confusion as to what the subject in focus might be.So my question is, do you feel it would be better to have a 2 camera rig set up side by side with a forced perspective and a full true 3d experience where the filmmakers it telling where to look or have a single 360 cameras with the ability to look where you want but maybe have the 3d effect diminished somewhat?This new vr concept is sort of the wild wild west and I'm looking to provide the best experience I can.My impulse is to do the 2 camera setup foe max 3d effect with some sacrifice on the head tracking.

This is a fairly popular topic.There are some good posts about this topic here:viewtopic.php?f=26&t=2357Also, some great posts about 360-degree cameras (with binaural sound) for filming Rift movies (with lots of sample videos) over at MTBS3D:http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=17378And more relevant threads at both websites too, if you use the forum search functions.

"geekmaster" wrote:This is a fairly popular topic.There are some good posts about this topic here:viewtopic.php?f=26&t=2357Also, some great posts about 360-degree cameras (with binaural sound) for filming Rift movies (with lots of sample videos) over at MTBS3D:http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=17378And more relevant threads at both websites too, if you use the forum search functions.Thank you so much geekmaster for the links.I had followed these threads for a while. Very insightful stuff. I guess the question I pose to you and the community is would you rather have head tracking or would you rather have a more immersive 3d experience where you are forced into a perspective?

"anthonybackman" wrote:Thank you so much geekmaster for the links.I had followed these threads for a while. Very insightful stuff. I guess the question I pose to you and the community is would you rather have head tracking or would you rather have a more immersive 3d experience where you are forced into a perspective?The immersive 3D experience requires head tracking, so you cannot really choose between them. Of course, we want both.

I've thought about this, and I think it is possible to have both 360 degree vision and stereoscopic, but it won't be easy. I think the camera technology to do this would require more technical development than the rift itself.If you think about the fact that when you turn your head, and the head rotates about the upper vertebrae, the position of the eyes changes and so there are slight changes in what can be seen (e.g., keep your eyes fixed in front of you, but turn your head - the objects close to you will move relative to objects further away). So essentially, we need a different 3D 360 image for each possible position of the head... so, infinite images, each taking a large amount of memory? Or at least finite, but with very small increments?Maybe it would be easier to generate a 3-dimensional model of the area around us, and texture it with data from our camera. Then we could calculate these minor changes in perspective. This still leaves something to be desired, because if we only have a texture from a single camera perspective, then when we shift our head, there will be outlines of new areas that we can see but for which we don't know what is there (i.e., from a photograph of a table, there is no way to determine what exists in the area that the table is occluding from the camera). Also, how do we generate the 3d model? Some sort of super robust 360 degree Kinect-style depth map that can be used in any lighting? Plenoptic cameras? Super-fast laser depth mapping? And how complex would the model have to be to handle things like translucency and reflection? Perhaps as long as we can get a single perspective 3D 360 image, it is sufficient immersion without calculating for minor changes in eye position (as though the camera is rotating about a single point as we look around)? Especially as long as there isn't anything really close to the camera.... but would that limit the types of experiences we could capture?Anyway, I'm not saying I have any answers, but I still think it's possible.

We just need a spherical light-field plenopticon camera that uses "fly eye" microlens arrays and a ball of image sensors, and a similar microlens array for our HMD as recently demonstrated by Nvidia (with the added bonus of variable focus accommodation in addition to stereoscopic vergence).Or we can use methods I posted in other threads, involving 12 fisheye cameras with heavy pre-processing, or two spherical cameras with light (real-time) video stream processing, for Rift DK compatibility. These lack depth-of-field lens accommodation, which limits 3D perception for some people.I have posted specific details for all of these methods, with plenty of links, in other threads.

Filmmaking for the Rift | Meta Community Forums

16 Replies

geekmaster
Protege
12 years ago
This is a fairly popular topic.

There are some good posts about this topic here:
viewtopic.php?f=26&t=2357

Also, some great posts about 360-degree cameras (with binaural sound) for filming Rift movies (with lots of sample videos) over at MTBS3D:
http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=17378

And more relevant threads at both websites too, if you use the forum search functions.
anthonybackman
Honored Guest
12 years ago
"geekmaster" wrote:
This is a fairly popular topic.

There are some good posts about this topic here:
viewtopic.php?f=26&t=2357

Also, some great posts about 360-degree cameras (with binaural sound) for filming Rift movies (with lots of sample videos) over at MTBS3D:
http://www.mtbs3d.com/phpBB/viewtopic.php?f=140&t=17378

And more relevant threads at both websites too, if you use the forum search functions.

Thank you so much geekmaster for the links.

I had followed these threads for a while. Very insightful stuff. I guess the question I pose to you and the community is would you rather have head tracking or would you rather have a more immersive 3d experience where you are forced into a perspective?
geekmaster
Protege
12 years ago
"anthonybackman" wrote:
Thank you so much geekmaster for the links.

I had followed these threads for a while. Very insightful stuff. I guess the question I pose to you and the community is would you rather have head tracking or would you rather have a more immersive 3d experience where you are forced into a perspective?

The immersive 3D experience requires head tracking, so you cannot really choose between them. Of course, we want both.
fakeblock
Honored Guest
12 years ago
I've thought about this, and I think it is possible to have both 360 degree vision and stereoscopic, but it won't be easy. I think the camera technology to do this would require more technical development than the rift itself.

If you think about the fact that when you turn your head, and the head rotates about the upper vertebrae, the position of the eyes changes and so there are slight changes in what can be seen (e.g., keep your eyes fixed in front of you, but turn your head - the objects close to you will move relative to objects further away). So essentially, we need a different 3D 360 image for each possible position of the head... so, infinite images, each taking a large amount of memory? Or at least finite, but with very small increments?

Maybe it would be easier to generate a 3-dimensional model of the area around us, and texture it with data from our camera. Then we could calculate these minor changes in perspective. This still leaves something to be desired, because if we only have a texture from a single camera perspective, then when we shift our head, there will be outlines of new areas that we can see but for which we don't know what is there (i.e., from a photograph of a table, there is no way to determine what exists in the area that the table is occluding from the camera). Also, how do we generate the 3d model? Some sort of super robust 360 degree Kinect-style depth map that can be used in any lighting? Plenoptic cameras? Super-fast laser depth mapping? And how complex would the model have to be to handle things like translucency and reflection? Perhaps as long as we can get a single perspective 3D 360 image, it is sufficient immersion without calculating for minor changes in eye position (as though the camera is rotating about a single point as we look around)? Especially as long as there isn't anything really close to the camera.... but would that limit the types of experiences we could capture?

Anyway, I'm not saying I have any answers, but I still think it's possible.
geekmaster
Protege
12 years ago
We just need a spherical light-field plenopticon camera that uses "fly eye" microlens arrays and a ball of image sensors, and a similar microlens array for our HMD as recently demonstrated by Nvidia (with the added bonus of variable focus accommodation in addition to stereoscopic vergence).

Or we can use methods I posted in other threads, involving 12 fisheye cameras with heavy pre-processing, or two spherical cameras with light (real-time) video stream processing, for Rift DK compatibility. These lack depth-of-field lens accommodation, which limits 3D perception for some people.

I have posted specific details for all of these methods, with plenty of links, in other threads.
fakeblock
Honored Guest
12 years ago
A sphere of outward pointing plenoptic cameras was the conclusion that I came to last night after some self-reflection following my post. I once read as a child that in a hologram, each point in the image contains data to completely replicate a 2-dimensional version of the object. In a similar vein, if we can capture enough data, and using the techniques involved in plenoptic rendering, we should be able to calculate a 2-dimensional image at each point (or at least, a large number of points, with interpolation in-between) around the sphere. Thus, calculate a 2-dimensional image for each eye separately, based on the head position and voila!

HOWEVER, since our heads pitch and yaw about different vertebrae (not to mention roll... or tilting your head, if we abandon the flight dynamic metaphor), our eyes can't always be placed neatly on a sphere. So there are still kinks to be worked out... Then again, when I go to the movie theater and watch a 3D movie, I don't even think about the fact that I can't tilt my head and still see the 3D effect, so maybe this isn't that big of a deal.
panmaster
Expert Protege
12 years ago
it is possible to record stereoscopic 360 deegre panorama which can be used for occulus rift..if you want to do it you have to have 3D-OMNICAM or build one on your own. for more information wirte to pan.master@interia.pl
polygonwindow
Honored Guest
12 years ago
Id argue that 360 2d content is 'good enough' now and should be focused on by filmmakers rather than trying a crazy complex 3d 360 setup. That will come. Right now in my experiments with 2d 360 video you still get a feeling of motion and depth (though obviously not ideal). Filming and watching a movie on the Rift requires a very big re thinking of cinematic language to tell a story properly. It will take many filmmakers and alot of experimentation to get it, but i believe we can certainly get it. Cinema took decades to evolve, and perhaps this new way of making a film will too.

Whats more important now is HOW you film with a superwide FOV rather than 2d vs 3d. How you arrange the shot will dictate where people will want to look, and the art and skill in directing an audiences 'look' will be what differentiates good 360 filmmakers from bad. Directional sound will help alot in this regard as well. Id say for any filmmakers reading this, do what I am doing...get a 360 camera setup and start shooting stuff. I can't do this alone! hahaha
candiedbug
Honored Guest
12 years ago
Something like this perhaps: http://actualites.epfl.ch/presseinfo-com?id=983

I also remember seeing a project on youtube that used 4 pairs of 360 degrees mirrored-dome cameras and some software trickery to generate two seamless panoramic videos, one for each eye. Can't remember the name of it though.
geekmaster
Protege
12 years ago
"candiedbug" wrote:
Something like this perhaps: http://actualites.epfl.ch/presseinfo-com?id=983

I also remember seeing a project on youtube that used 4 pairs of 360 degrees mirrored-dome cameras and some software trickery to generate two seamless panoramic videos, one for each eye. Can't remember the name of it though.

They applied for a patent for sticking a bunch of cameras looking outward from a sphere or hemisphere? Since when is that novel enough to deserve a patent?