Augmented Reality

http://www.indiegogo.com/projects/vision-sensor-creates-new-seeing-capabilities-for-the-oculus-rift/

THIS. THIS. THIS.

Maybe I'm missing something, but VR + corresponsing external lenses/cameras/visions seems to be hands down the ultimate solution for allowing true and complete augmented reality...

Please allow me to explain my thought process.

AFAIK there are 3 basic methods which for implementing augmented reality ( these can be combined obviously )

1. Projection / Projection mapping strategies ... images are projected onto reality , to further the effect user may wear lenses w/ various types of filters to enhance certain spectrums etc.
MAJOR LIMITATION: when designing projected images and effects we will have to account for nearly infinite amount of different "natural lighting" scenarios... this may be worked around controlling users environment / lighting... but for me this is an unsatisfactory limitation.

2." Google glass-esque " - User wears transparent/ semi transparent lens and images are projected onto lens. Now we only have to worry about images will interact with actual as opposed to the many varieties in (1) .
BUT...now we have to deal with not only projecting onto translucent lenses, but how images on the lense will alter the users "view" of reality.
Many work arounds here, but ultimately it seems like this will limit the ways we can mix the users view or reality and view of images , unless you add an external camera, which leads to...

3. computer generated image + image from external camera. This set up is much simpler and "mixing" of computer generated view + view of reality. All "mixing" is done via computer so we don't have problems in scenario (1) ( i.e. accounting for real world physics when projected images onto surfaces / lighting interactions ) ... and we dont have to worry about optimizing images for translucent lenses like in scenario (3) .

So if scenario 3 is the best implementaion.... does it not follow that we VR + Corresponding external cameras is the most powerful way to implement this !?!

the VR gives us complete 360 virtual world

The external cameras gives us a nearly perfect view of reality.

SO... with this type of implementation are we not completely free to mix any type of computer generated images with anything we could see in reality ?

isn't the only limitation here our ability to be creative and design anything we choose ?

Am I missing something here ? ... Are there any reasons NOT to use this implementation ?

I think the Oculus Rift is great, but IMHO oculus rift + corresponding external cameras allow for SO MANY more possibilities and uses that we shouldn't even be considering the Oculus WITHOUT this feature !

Excuse my apparent hyperbole, but it just seems like adding external cameras to the Oculus is like hooking connecting a computer to the internet in that allows for a complete paradigm shift that is VASTLY more powerful....

Hope I inspired somebody, peace out.

"grayzen" wrote:
AFAIK there are 3 basic methods which for implementing augmented reality ( these can be combined obviously )

1. Projection / Projection mapping strategies

2. "Google glass-esque " - User wears transparent/ semi transparent lens and images are projected onto lens

3. computer generated image + image from external camera

Hello grayzen. Nice to see that someone is very enthusiastic about AR :)

What you have described here is little bit mixing the different aspects of AR.

Point 1: Projection based AR works very well for abstract AR information such as highlighting features of the real world, attaching text or symbols to objects etc. You are spot on with the problem of lighting conditions. This is huge challenge for projection based AR. For concrete AR this is very limited approach in the classical sense of visualizing 3D objects and producing such to the real world as if they were real objects. Even though it has inherent advantage of showing the AR content on the objects. VALVE spinoff technical illusions and CastAR is quite nice approach for attempting solve the problems, despite it having major practical limitations (Could be interesting to see spherical "cave" of reflective material used with CastAR for pure VR experience).

Point 2: Optical see through based AR is propably most accurate term for what you describe. Now, people should realize that Google Glass is not truly classical AR solution. To describe GG critically would be saying it is only floating smartphone screen on your field of view, basically saving you from pulling your phone off your pocket. Thats little pushing the limits but this is the main value proposition what GG has. Now, GG is very suitable for abstract AR, it is not such a joke of product. But when we actually start speaking for overlaying content on your field of view, we are entering the realm of conrete AR. Here you need Oculus Rift style of product that has clear optical path between the user eyes and real world, where inbetween there is display device that is capable of producing the visual content. There are not so many products that actually deliver this. There is Silicon Micro Display ST1080 and Meta1 that is based on Epson Moverio. While none of them actually is able alone to give us the "dream" what we all have for AR, those get closest. I dont want to go into too much details why this approach is not practical or technically feasible. Some hints of it you can get from Michael Abrash blogs.

Point 3: Now you are absolutely right that this is closest you can get for experiencing the AR what we all dream of. But here we are starting to replace our most precious sensors, eyes, with pitiful cameras. Delays, low resolution just to say the most obvious problems. This is the very nature of all those fake AR videos and "visions of AR" what we have seen since 1984 after terminator movie popularized the technique. What really is happening is good combination of perceptual tricks that trigger our brains to believe things are 3D. Even if we see them on a 2D display (For reference see Johnny Lee head tracking video to understand). While this kind of video see through does deliver the closest of AR experience, the problems what we get from replacing our vision with cameras to see the world are too big tradeoffs for the time being. Who knows if we come up with nice tricks to remove this lack.

If you are confused for what I mean by Abstract AR and Concrete AR, please see the following lecture on the topic:

What is Augmented Reality and Why it emerges?

Keep up the enthusiasm. There are a lot of us who are working hard to realize those dreams :)

- Sauli

"Cromfel" wrote:
What really is happening is good combination of perceptual tricks that trigger our brains to believe things are 3D. Even if we see them on a 2D display (For reference see Johnny Lee head tracking video to understand). While this kind of video see through does deliver the closest of AR experience,

Oh man that Johnny Lee video just sent me down memory lane.

It's funny, because that demo is super impressive when you watch it on youtube- but the reason its actually impressive is because Johnny himself and the world he's filming himself in are 2D as well! .. So the projected world appears to be the same as Johnny's actual world when you watch the demo.

If you set it up yourself, it's nowhere near as impressive.. because the screen is 2D, and you're actually moving around and viewing a 3D world.. A big surprise for me once I finally got it working.


I really think everyone is going to be blown away by AR on the Rift when it finally arrives. Most AR solutions today rely heavily on image processing or environment scanning which is just awfully crappy and inaccurate by default-

If we have Oculus AR that uses image processing just to initially "place" the objects on a single objects (which can happen for any duration of time, and then rely on Rotational and Positional tracking to move the imagery at 60fps.. that's going to be so incredibly superior to any other AR platform in development.. not to mention extremely easy to develop for.. The only issue I foresee is the rotational drift.. but that might even be a blessing in disguise because you could use the camera data to subtlety correct it.

OVRs sleeves are loaded with ACES. =)

"antigravity" wrote:

It's funny, because that demo is super impressive when you watch it on youtube- but the reason its actually impressive is because Johnny himself and the world he's filming himself in are 2D as well! ...

...If you set it up yourself, it's nowhere near as impressive.. because the screen is 2D, and you're actually moving around and viewing a 3D world.. A big surprise for me once I finally got it working.

Exactly. And this is the one trick pony most of the impressive demos we see on youtube are riding with. For example Meta1. I would be really happy to see functional optical see through AR. But given the technical challenges ahead on that front it just is not going to happen for a while.

"antigravity" wrote:

If we have Oculus AR that uses image processing just to initially "place" the objects on a single objects (which can happen for any duration of time, and then rely on Rotational and Positional tracking to move the imagery at 60fps.. that's going to be so incredibly superior to any other AR platform in development.. not to mention extremely easy to develop for.. The only issue I foresee is the rotational drift.. but that might even be a blessing in disguise because you could use the camera data to subtlety correct it.

OVRs sleeves are loaded with ACES. =)

The nice thing about video see through AR is that the latency of camera or tracking does not really hinder the AR experience that much. Since the content you see is all delayed in the same amount. So everything is equally "crappy", if you will, making the experience somewhat uniform with image analysis based tracking solutions such as Vuforia (Actually using very low latency tracking breaks the immersion compared to one that is 1:1 delayed). While it looks quite good, practical applications outside of amusement are also little bit further away. Maybe utilization for entertainment will give a lot of freedom on it. But serious applications or lets say wandering on the streets engulfed with OVR might be tad dangerous :)

- Sauli

"Cromfel" wrote:

The nice thing about video see through AR is that the latency of camera or tracking does not really hinder the AR experience that much. Since the content you see is all delayed in the same amount. So everything is equally "crappy", if you will, making the experience somewhat uniform with image analysis based tracking solutions such as Vuforia (Actually using very low latency tracking breaks the immersion compared to one that is 1:1 delayed).

It's true- Vuforia looks decent. But it's probably good to recognize that the delay you see probably isn't caused by any sort of camera latency. For instance if I turn on my video camera straight on my HTC-one, and shake it around- I don't see any noticeable latency whatsoever.

It's really the image processing that slows it down building up a display queue.
Video frame in -> process image based AR (delay) -> Video frame out.

If you have an AR solution that's build on realtime rotational and positional data from the OVR unit, you get to skip that processing step all together providing a quality realtime AR experience.

"Cromfel" wrote:
While it looks quite good, practical applications outside of amusement are also little bit further away. Maybe utilization for entertainment will give a lot of freedom on it. But serious applications or lets say wandering on the streets engulfed with OVR might be tad dangerous :)

Hahhahaa.. Agreed. Man, I watched the VRase promo video and they had someone riding around on a skateboard with the unit. Complete ridiculousness in the name of rushed hype.

While I have no idea what direction OVR is headed I'm assuming they are focused on the static-player experience.. As in for the near-term you're tethered to your computer and hardware. In which case it's the seated and static AR experience that I think will be the first focus. And frankly until that's done perfectly- it's awfully premature to stick people on skateboards!! ;)

"antigravity" wrote:

"Cromfel" wrote:
If you have an AR solution that's build on realtime rotational and positional data from the OVR unit, you get to skip that processing step all together providing a quality realtime AR experience.

Well, I was specifically mentioning video see through for a reason. I actually ran some tests using setup with image based tracking, that was working on a reasonable level at 30fps, then did the same test with optical tracking system from Vicon that was pushing out the tracking data at 120Hz. That's what I meant for the difference. Since the video see through was pretty much at its limit, combining that with good tracking system did not make the experience any better, quite obvious and expected, because the augmented content was moving faster than the video background. So advantage of good tracking was wasted as the "reality" in this case was limited to the camera specs to get video-see-through AR. Kind of "out of the box" advantage for video based tracking within the device for video see through.

Whereas the first system was inherently syncing the AR content with the video feed perfectly, hence the user experience was "ok". As the tracking was based on the very same feed as the user saw. So to re-iterate: Having the tracking in perfect sync with the visual content of real environment is more important than the speed of sync or update in case of video see through applications. Where for optical see through applications the faster the tracking the closer you are to your natural perception :)

So you are absolutely right, having faster tracking system is always better, given the condition that you can sync the video see through of reality with the tracking data properly. So you are bound to the limits of your camera FPS plus the problem of syncing the tracking with it. Nothing special but just interesting realization that gives some bonus for the solutions such as Metaio/Vuforia/Alvar.

"antigravity" wrote:
While I have no idea what direction OVR is headed I'm assuming they are focused on the static-player experience.. As in for the near-term you're tethered to your computer and hardware. In which case it's the seated and static AR experience that I think will be the first focus. And frankly until that's done perfectly- it's awfully premature to stick people on skateboards!! ;)

I really hope they intended that for provocation! :twisted:

So what exactly are we augmenting?
I'd love to see electric fields when working on industrial machinery, but otherwise it has to partially obscure the daylight view of the scene.
Thermal imaging is useful for low/no light conditions, but ir emissivity works differently than reflected light and can be deceiving. Thermal is also ridiculously expensive, like $10k for 640 x 480, but again useful for air conditioning work.
RF energy would be cool to see for a bit, but there is so much around us it woulds probably be overwhelming.
I do not want ads overlaid on my sight. If you want it for texting, go diaf.

"Roaster" wrote:
So what exactly are we augmenting?
I'd love to see electric fields when working on industrial machinery, but otherwise it has to partially obscure the daylight view of the scene.
Thermal imaging is useful for low/no light conditions, but ir emissivity works differently than reflected light and can be deceiving. Thermal is also ridiculously expensive, like $10k for 640 x 480, but again useful for air conditioning work.
RF energy would be cool to see for a bit, but there is so much around us it woulds probably be overwhelming.
I do not want ads overlaid on my sight. If you want it for texting, go diaf.

Very interesting use case you brought up Roaster. This is exactly the kind of thinking we need to actually make use of Augmented Reality. From current technology obsession to actually trying to wrap our heads around what kind of information we could actually utilize from concrete reasoning side.

How about your google glass indicating to you when sound levels exceed harmful dB? Yet another of those things we are quite poor at making judgement of. Maybe such apps already exists but in any case..

I wonder how helpful it would be to use some of the sports tracking equipment for heart rate, oxygen levels or whatnot constantly being shown to you through such device as Glass. For example when training to long distance running and trying to push your body to limits without exhausting your self, in the sense of building better understanding of your body through constant feedback loop that can be interpreted easier through abstract means than just based on feeling.