Spatial Video on the Quest

[Hiro_Protag0nist]

OK I'm a big fan of all immersive video/media so I'm really happy that spatial video is now on the Quest 3 and hopefully soon we'll be able to have it on there with the click of a button from our iPhone 15 Pros - nice!

But, I've watched videos from very respected videographers saying spatial video is so much more than 3D and after I watched the demos yesterday I'm struggling to see the difference.

Could someone tell me what the difference is to the viewer (don't say the edges are blurred)?

What are Apple doing to make this difference?

If you know of any good technical articles online please point us to them 👍

[LaserDisk]

I don't have any technical articles, but what I've observed is a heightened sense of depth. For instance, when I watch a 3D movie, the screen is still flat and some effects pop out. However, with spatial video, the images seem to come alive. It's like I can almost immerse myself in the video and be a part of that world. I hope I'm making sense here, but honestly, it's hard to put into words.

[Hiro_Protag0nist]

Hi @LaserDisk , thanks for the reply.  I don't think I'm seeing any heightened sense of depth (I will of course check again).  If you watch some good quality 180 3D content from people like Hugh Hou (make sure you download it first, don't stream), surely it's the same? 

If not, then I should think standard 3D content could also get this sense of depth in-post?  One good thing I noticed is the small file size.

[Hiro_Protag0nist]

I'm certainly not hating on spatial video, as I said, I'm a big fan of immersive media and am happy that I can now use my phone to record it.  
Just did a quick search and found this.

[Hiro_Protag0nist]

OK so I'm beginning to wonder if things look different if recorded by the AVP vs iPhone.  Here's a video by Hugh Hou:

I wonder how the samples on the Quest were recorded?

[DaftnDirect]

I was looking into this a few days ago and from what I can find, spatial video is purely 3D stereo, nothing more than that. Apple give it the 'spatial' treatment cos it sounds good.

The iphone 15 uses 2 of it's forward facing cameras to shoot the video and the only reason other iphones can't do that is because the 15 is the only one with the movie capable cameras positioned in-line.

The result is 1080p 30fps 3D left+delta encoding at 251 Mb/s, similar to Blu-ray 3D, but using MV-HEVC codec in a MOV container. So if you view on a mono device, it ignores the delta and just shows the left view.

My DVDfab converter can convert any 3D vid to (and from) this format, no problem and I suspect most converters can do the same.

the only difference seems to be the way the vid is presented... in a fuzzy edges shadow box of some sort, but the vid itself is just 3D.

[jay.arg]

Hi, I'm also testing Spatial Video with the V62 feature, but I've been experimenting with it before by converting videos to stereo videos using an app that converted iPhone spatial videos into HOU so I could watch them in Skybox. I've also been recording 3D video since 2019 using a special camera with two lenses. My opinion is that the quality and vividness of the colors of the videos recorded by the iPhone are really impressive. The only bad thing is that the FOV is limited, very limited. My 3D camera recorded 180 degrees, so it was really like putting your head in the image. The bad thing was that the colors were dull and the quality was pretty bad despite recording in 5K. At the moment, the big difference I see between the spatial videos recorded and uploaded with the V62 and the conversion I did before is that the FOV is reduced even more by that blurring they put on the edges, which I hope can be turned off in the future.

To answer your questions, the difference between spatial video and 3D video is that spatial video uses a new format called MV-HEVC, which automatically adjusts the depth of the scene and reduces the storage required for 3D files. Apple claims that spatial video is more immersive and realistic than 3D video, but some people have challenged that claim and said that spatial video is just a fancy name for 3D video. Apple is using the iPhone 15 Pro's dual cameras and Vision Pro's display to create and show spatial video, but it is not clear if they are using any other sensors or technologies to enhance the effect. 

One issue with the iPhone cameras is that they are very close together, so they don’t capture a good sense of depth like my 3D camera or the Apple vision Pro cameras, which are spaced apart like human eyes. However, I have found that taking a physical step back from the virtual screen where the video plays helps to improve the depth perception of the spatial video. You can try this tip next time you watch a 3D video and see if it works for you.

[DaftnDirect]

I think the only manipulation that's being done (if any) on the raw footage is computational adjustment to compensate for the two cameras being closer together than your eyes. Some people are saying that's being accomplished by adjusting the zoom and crop, some people saying lidar being used to compensate but either way, the effect is to replicate using a stereo camera with the correct lens separation. I can't quite decide from the example footage I have whether it does this perfectly or it's more of a half-way house between mono and standard 3D.

Either way it's great marketing.

[Hiro_Protag0nist]

Agreed.  Even if it's some new amazing type of video, I don't see it with my eyes.

I'm just going to look on the bright side and be happy that it's millions more people with a 3D spatial 😁 camera.

[DaftnDirect]

Yep, there's definitely nothing new.

I would assume the cameras in the Vision Pro headset have the correct separation for 3D recording but it's the same as the Quest 3, there must be compromises in video quality when the priority has to be tracking frequency, accuracy, low latency... and power consumption. The iphone will have better cameras for video but they're just close together.