Photogrammetry VR

[cubytes]

Hey Guys,

I am interested in Photogrammetry VR for three reasons;
-I like the idea of VR being almost like a teleportation device
-I like the idea of adding sound, VFX and subtle interactive elements to photogrammetry scenes
-I like the idea of presenting photogrammetry VR experiences to the public in an Art Gallery type venue

What is photogrammetry?

https://www.youtube.com/watch?v=B8FPunc_RTE

These types of Room Scale VR experiences are typically less about interactivity and more about immersion.

Realities.io looks like a great application for photogrammetry VR on Steam. The application is free and it appears to be intended as a Google Earth like platform for photogrammetry scenes. I think a platform that specializes in photogrammetry VR is great for producers of photogrammetry scenes and users who are interested in exploring neat places around the world in VR. Not sure if the scenes in Realities.io typically come with sound or not? 

Valve also bundled a photogrammetry scene in with their labs demo too.

I really like the idea of being able to explore neat places around the world in VR.

I also would like to experiment with adding soundscapes and background music to the experience either by directly recording audio from the environment during the scanning process or just adding in pre-recorded soundscapes that match the environment of the scene after the fact.

To add even more immersion, I would definitely like to experiment with adding digital VFX to the scenes. For a platform that specializes in photogrammetry the scenes themselves would come with metadata that specifies which VFX sets would work and make sense in the scene. Then the application would auto update, download and install a bunch of VFX and while the user is exploring said scene an algorithm would determine which VFX set to initialize and cycle through. Some of the VFX sets would also need to have sound that accompanies it. 

Interactivity

As far as interactivity goes the experience is more of a sight see and explore so any interactivity would be secondary to that primary use case.

One idea to add some level of interactivity to the mix is to create a sort of optional Easter Egg hunting activity. This would require developers to generate lots of virtual objects and hide them in various photogrammetry scenes on the Realities.io platform. The user would be tasked with exploring scenes and collecting these objects. So when exploring the scene you can use the controllers to hold up a "treasure book" UI that has an image of what you are looking for in the scene. The virtual objects can be picked up, thrown, and once you have found a particular object you can use the treasure book to spawn that object in the scene allowing the user to hide it somewhere and then be able to challenge their friends to find it.   

Then there is also the idea of picking up where Valve left off.

In that while exploring these scenes you could spawn one of many different virtual pets to accompany you. Some pets might only work on certain kinds of scenes. Naturally the virtual pet would come with an assortment of virtual objects and activities that go along with that pet. Like you could play catch with the pet or feed the pet, and basically level it up and have it help you locate and collect the Easter eggs.

And while on the subject of virtual characters being spawned in the scene I suppose one could create a virtual tour guide character too that can be spawned into the scene or removed from the scene at will by the user.

Photogrammetry VR in an Art Gallery




I think photogrammetry VR is kind of like the VR equivalent of an Art Gallery.

However the idea is to use the Art Gallery itself as a window into photogrammetry VR. So you would probably need a customized iteration of the Realities.io app that launches directly into a particular scene at a specific POV or be tasked with making make your own application and your own photogrammetry scenes for this use case.

So when a user approaches an exhibit they will be able to admire a photo hanging on the wall like they normally would in an Art Gallery. The idea is that when the user puts on the VR headset and enters into the photogrammetry scene they will start off at the exact same POV that the photo hanging on the wall has.

As always,

Food for thought

[nosys70]

well you mix several concept here.
photogrammetry is the use 2D picture to measure objects (or distances between objects).
photogrammetry VR is a 3D extension to make 3D reconstruction with some scanning device (lidar)
The result from photogrammetry VR are numeric values (cloud of points) .
Using textures shot from another camera, you can recreate the illusion of 3D objects (room or object)
but a cloud of points is not a 3D object.
The first problem is you need to to scan the place in order to avoid occlusion, or else you only get partial object data. That is easy for some objects like a statue, but even a simple chair will require a lot of work.
The second problem is that your cloud of point makes no difference between the chair and the carpet it is laying on. So at a moment, you need to convert your cloud of points to a mesh, and edit the mesh to separate object.
(you can also do that at the cloud level)

This can be relatively easily automated for some situation (assuming flat floor is an object, and things on floor are different objects) but it will definitively requires a huge editing, to separate books in a library for example)

So usually , what you see is a mix of technology, where spaces are scanned to get a rough idea of a volume, and usual object are replaced by real 3D object, so you can eventually interact with).

[cubytes]

nosys70 said:

well you mix several concept here.
photogrammetry is the use 2D picture to measure objects (or distances between objects).
photogrammetry VR is a 3D extension to make 3D reconstruction with some scanning device (lidar)
The result from photogrammetry VR are numeric values (cloud of points) .
Using textures shot from another camera, you can recreate the illusion of 3D objects (room or object)
but a cloud of points is not a 3D object.
The first problem is you need to to scan the place in order to avoid occlusion, or else you only get partial object data. That is easy for some objects like a statue, but even a simple chair will require a lot of work.
The second problem is that your cloud of point makes no difference between the chair and the carpet it is laying on. So at a moment, you need to convert your cloud of points to a mesh, and edit the mesh to separate object.
(you can also do that at the cloud level)

This can be relatively easily automated for some situation (assuming flat floor is an object, and things on floor are different objects) but it will definitively requires a huge editing, to separate books in a library for example)

So usually , what you see is a mix of technology, where spaces are scanned to get a rough idea of a volume, and usual object are replaced by real 3D object, so you can eventually interact with).

Interesting....

Perhaps I should have titled the post "Photo realistic VR" instead of "photogrammetry VR"?

Specifically I am interested in how photo realistic VR scenes are associated with Room Scale VR allowing the user to explore (walk/teleport) said scene. The use case is similar to 360 photos and videos but on a whole other level.
 
After reading your comment in effort to understand a bit of what you are talking about I briefly looked into photogrammetry on this blog post here

I am mostly interested in the use case itself and how to enhance it after the scans are complete.

But I would also be interested in the process of photogrammetry and learning how to use the tools involved with generating photo realistic VR scenes too. On the Realities.io site they say "Most of our scans are based on photogrammetry. We also use other scanning techniques like LiDAR when necessary." 

I mean who wouldn't want to travel around the world and do scans of interesting locations for this kind of use case? I know I would sign up for that in a heartbeat.
 
Realities.io also mentions:
"We just launched a pilot program in
which we work together with selected content creators from around the
world that are interested in scanning interesting locations. Do you have
a background in (panorama) photography and/or film-making, often end up
in cool & interesting places and are interested to create
photogrammetry content? Apply for our pilot program"

I realize that with photogrammetry you would be limited to well lit places with very little to no motion in the scenes for it to work. But even with that limitation you could still capture some interesting scenes for users to explore.

I think the Realities.io platform would do well to cater to a specific kind of photo realistic scene. A photo realistic scene that has a completely artistic purpose, or a scene that has historical significance and could be considered artistic yet informative.

But the point is that I would like to enhance the scans after the fact. In that I would like to experiment with adding layers of CGI and VFX to the scans as well as experiment with adding subtle interactive elements to the experience. 

I mean the idea is to leverage the Realities.io platform (which hopefully skews more towards artistic content rather then commercial and marketing content) to create a "treasure hunting game"

The other idea is to leverage the platform to create a public VR exhibit that provides an interesting transition from real life to VR designed specifically for photo realistic VR. So at an Art Gallery the idea is that users would be able to admire a gorgeous photo hanging on the wall (like they normally would at an art gallery) but then be able to put on a VR headset and walk around in a photo realistic VR scene starting off from the exact POV as the photo hanging on the wall has. Which I think would be a transition that really blows peoples minds.
 

[nosys70]

Microsoft got a project like this where they are able to recreate a place from the thousands pictures taken by people and posted on the web( https://photosynth.net/ ). So you can imagine that the capture work is pretty already done for most of touristic places.

You already have the people scanning archeologic site in Africa , to save them (digitally) from destruction because the war.

on the other hand, if you need a quick and dirty reconstruction, you can do that with with one of the many lidar camera available, like the Kinect, the primesens, the intel realsens etc....

For example , in a few minutes I can scan my living room and get a perfect 3d picture that I can use for a VR visit. (check programs like Kscan3D, Scenect or Reconstructme).

unfortunately as posted before, this kind of scan is pretty unable to differentiate objects, so if you just want to visit a room, it could do the job, but if you expect some interaction (opening a door, moving a chair) it is pretty useless.

The main problem with scanning vs 3d editing is mesh topology.

When you scan, you got a random cloud of point, that is reduced to a mesh (basically, reducing the number of point to a reasonable amount and then linking them).

so imagine you scan your arm, and get a very convincing result.

The problem is when you want to apply deformation to your arm (when you bend it).

in the reality your arm is bending particularly at the elbow in a particular way.

if you bend your digital arm, there is absolutely no chance it give a realistic result, because the mesh is giving no indication where and how it must bend, so it will likely bend like a garden hose at best.

actually, this is a very common issue and some people are working on it

https://3dideas.wordpress.com/tag/topology/

Retopolgy is a time consuming process, so most of the time it is faster to take a generic shape (like a human body) with correct mesh topology and edit it (you can use the scan as a guide)

there is a company doing it from two shots (face/profile) https://www.bodylabs.com/

https://www.youtube.com/watch?v=mzKHFLV_9jc

[cubytes]

nosys70 said:

Microsoft got a project like this where they are able to recreate a place from the thousands pictures taken by people and posted on the web( https://photosynth.net/ ). So you can imagine that the capture work is pretty already done for most of touristic places.

You already have the people scanning archeologic site in Africa , to save them (digitally) from destruction because the war.

on the other hand, if you need a quick and dirty reconstruction, you can do that with with one of the many lidar camera available, like the Kinect, the primesens, the intel realsens etc....

For example , in a few minutes I can scan my living room and get a perfect 3d picture that I can use for a VR visit. (check programs like Kscan3D, Scenect or Reconstructme).

unfortunately as posted before, this kind of scan is pretty unable to differentiate objects, so if you just want to visit a room, it could do the job, but if you expect some interaction (opening a door, moving a chair) it is pretty useless.

The main problem with scanning vs 3d editing is mesh topology.

When you scan, you got a random cloud of point, that is reduced to a mesh (basically, reducing the number of point to a reasonable amount and then linking them).

so imagine you scan your arm, and get a very convincing result.

The problem is when you want to apply deformation to your arm (when you bend it).

in the reality your arm is bending particularly at the elbow in a particular way.

if you bend your digital arm, there is absolutely no chance it give a realistic result, because the mesh is giving no indication where and how it must bend, so it will likely bend like a garden hose at best.

actually, this is a very common issue and some people are working on it

https://3dideas.wordpress.com/tag/topology/

Retopolgy is a time consuming process, so most of the time it is faster to take a generic shape (like a human body) with correct mesh topology and edit it (you can use the scan as a guide)

there is a company doing it from two shots (face/profile) https://www.bodylabs.com/

https://www.youtube.com/watch?v=mzKHFLV_9jc

I see what you mean now.

In terms of interactivity I wasn't thinking about anything in the scan itself being interactive.

I was thinking more about hiding virtual objects in the scenes to create a "treasure hunting" type of meta-game. Only the virtual objects would be interactive.

That and also the idea of having virtual pets and virtual tour guide like characters in the scenes.

Adding Multiplayer like features.... 

And another idea would be to add some tiltbrush like stuff to the experience. Then that would allow users to do virtual graffiti and what not.

All of those ideas are secondary features that should improve overall user engagement with the content. I mean there is only so much you can do with the content given the use case. It is not a game, this use case is more like presenting VR as a teleportation device that allows you to walk around a bit in Room Scale. It is similar to 360 photos and videos but on a whole other level and with its own limitations.  

It will be interesting to see how the Realities.io main menu UI holds up to a vast amount of content.

It would also be interesting to see different UIs for specific collections of scenes.  

I really would like to add CGI and VFX to the scenes as well but more as a finishing touch though. I mean the use case is all about photo realism so you wouldn't want to go too crazy....

These are all ideas for features that could either be built into the Realities.io application or used to create your own application/platform to compete with Realities.io.

[nosys70]

Easypano tourweaver is exactly this.
you take a 360 picture from a place and can add active content (object , or video on a tv set, or dynamic picture on a wall , or sound when you get closer to an object etc..).
If you need the same with ability to freely move and interact with environment, you can do the same with unity 3d.
your 360 picture (or video) will be a skybox, you 3D scan can be the close environment, and you can ad as much as props (object, characters, vehicle, furnitures) as you want from the asset store, including grass or clouds.
https://www.youtube.com/watch?v=39cLOJlhhTg

at macro level , you can easily import real terrain to unity.and just place real 3d building or add grass and trees
https://www.youtube.com/watch?v=QZ6nlq-uvnw

[jademan]

Article out about new Valve software "Destinations" to make photogrammetry more accessible to all.  Just saw it and thought it would be appropriate here for anyone just starting to learn more about it - like me!
http://www.roadtovr.com/valve-releases-destinations-a-free-vr-creation-tool/

[edmg]

Yeah, I just checked out the new Destinations app. It's very cool, some of the scenes look very real.

[nosys70]

yep but see the price...
the problem with pure photogrammetry (taken from plain 2d photos), is the huge number of pictures required, and the issue you could get to capture a crowdy place while taking dozen of pictures.
And you cannot check the result without huge/long data processing.
Add to that, that most of VR glasses offer poor resolution that is not able to render textures in high quality.
Lidar-like capture gives you the 3D at the same time as capture and almost in realtime.
The only drawback of Lidar-like camera is they are based on low resolution camera, so the textures are usually poor, but still acceptable for that current VR screens.
the workaround is to merge a lidar-like camera and a good DSLR like this
http://jamesgeorge.org/RGBDToolkit
http://www.depthkit.tv/

[kernow]

nosys70 said:

the problem with pure photogrammetry (taken from plain 2d photos), is the huge number of pictures required, and the issue you could get to capture a crowdy place while taking dozen of pictures

You can minimize this particular issue with better quality photos. The higher the resolution (makes it easier to do the point cloud calc); clarity of the image (spacial resolution and lack of DoF blur, lack of motion blur, all again helps with calc); and good, bright, even lighting with no specular highlights or artifacts (gives less sensor noise to mess with point cloud calc, and better quality textures); then the fewer images you will need. Most consumer lenses and camera sensors (well, pretty much all) are not ideal for photogrammerty, and the worse the photo images are, the more photos will be needed and more clean-up needed once you get your point cloud/mesh. This means, you need to at least go into prosumer-level lenses (try to go with primes rather than zooms) and sensors, ideally at or above 36Mpixels, if you want to minimize the number of photos and minimize post-calc clean-up.