Waiting is hard... (Ideas while we wait)

"donkaradiablo" wrote:
I think I know what's going to be the next big thing.

Real 3D "movies" of real people that you can watch from all angles as if they were there with you, with positional tracking, captured with multiple scanners, played back thanks to Carmack code compressing the hell out of that 4D data to be accessed and streamed to the GPU as needed depending on where you look.

Not 3D 360 degree movies... not real time rendered fake stuff. Real people, like they are really there with you.

Not whole stadiums, just one person. Also movies as they scan actors and put them in CGI spaces anyway. And as we take our loved ones to be scanned so we get to keep their memories forever, maybe we'll get that feeling our grandparents must have felt when they took our parents in for their first photos.

No physics, no dynamic lighting, no simulations, no artificial intelligence, no destructible environments... just predetermined, recorded stuff streamed using the information of where you are looking and the prediction of where you will be looking, which Oculus is already doing.

Yes, it's a whole lot of data. But all types of work andoptimizations Carmack has done and let us know he looked into in the past years should translate so well to this. And he has supercomputers with special purpose computing units at his fingertips, with lower level access.

What we have seen so far is nothing. Knowing that is what makes waiting hard. It's like you knew the photo was coming to town and everyone is excited but you know they are doing this thing called "movies"and you believe it will be big.

(edited: subject)

this is what i actually thought VR Cinema was when i first got my DK1. boy was i disappointed.....but not really. many other games/demos were still fantastic with VR.
i also think that was you have described is the future of movies. not watching a movie, but BEING IN the movie. free to move around the scene, and watching it from any angle....that would be amazing.
not sure how they would shoot such a movie, but it's basically the same thing as a role playing video game. take Skyrim for example. it's telling you a story, but with VR you are INSIDE the events, and you are even interacting with the story.
again, not sure what kind of cameras you'd need to render the space on a 360 degrees, but it's very doable i'm sure. it would probably have to be something like those 3D multy light scans from Veiviev, except you'd be capturing motion as well. actually Veiviev already captured very basic motion (breathing) and it looks fantastic.
This is very doable. it just takes someone with money that wants to shoot a movie exclusively for VR. to get it doe right, you have to have VR in mind from the start.
someone should really try and make a short demo. Neo learning to navigate the Matrix from Morpheus should be fitting theme for such a demo :D.

Nowadays, the way they shoot movies is actors in green screen and location in CGI. Even parts of actors are CGI, like in movies like Guardians of the Galaxy, Avengers or Man of Steel. So as long as they capture the actors performance in 4D, fourth dimension being time, meaning it's not just models from still shots of the subject with depth information but a continuous capture of the motion of the subject with depth information, they can place it in a CGI environment. Basically what they have been doing in 2D, but now in 3D.

In VR, the environment can be rendered in real time and those 4D captures might be dropped in that world. What happens today is you have one model with data consisting of of polygons, textures, just one and that is the data needed. Animation is applied to it and you have an animated 3D object. For 4D capture to work, the engine would have to stream new data every frame, data of a full capture of the subject at that frame.

Today, when they scan objects to put them in the game, they optimize that content by converting all that data to lower polygon models and normal maps. Similarly game content creation tools might optimize the hell out of that 4D data. And just the way megatexture tech works, streaming in data required for where you are and what you are looking at, those 4D captures if turned into voxels, could have the same logic applied to them, kinda like in
this demo. Or those 4D captures could just be converted to 3D models and animation data, like in this one. Or maybe that 3D model and animation data, could be used as a shading layer, over that 4D capture of the subject, having the best of both world, real performance with a layer of lighting that matches the environment. Instead of voxels for the high fidelity capture layer, this could also be used. Voxel data at low res could be used for the lighting pass, the way nvidia does for global illumination with VXGI. Determining whatever combination of these works best is the kind of experiments I would hope the Carmack we know and love would be doing at Oculus .

So in the end, the solution would be a hybrid form of rendering,
1. The world around you rendered in real time just like it is now in UE4 archvis demos like these: 1, 2, 3, 4, using laser scan captured assets from the real world like in this one.
2. Simple Stereo3D video based sprites used for people far from the viewer, like the people in the distance in this one between 0:30-1:00
3. Objects and NPCs at medium distance from you, photoscan based, animated with physics applied to them. Like the Vieviev models, or Quantum Capture models, or a table that gets knocked over during the scene.
4. Closer to the viewer, it's 4D captures, in voxels, displayed at the highest resolution if you are looking at them, blocky if you are not or parallax capable movies like PresenZ-VR, with a layer of shading/lighting applied to it to blend with the environment. It's non-interactive emotional content.
5. Mobile in the environment, objects and NPCs at varied distance from you are photoscan based, animated and have physics applied to them. They also have glasses, masks and whatever so they do not take you to that famous uncanny valley.
6. Models that are a mix of all these. Thing of Barney in HL, being a real person under there when he takes of the mask, when the action starts and he puts on the mask, it's an animated character. Your brain building it's perception of reality based on pattern recognition, prediction, merging memory and what's happening in the moment, would blend those two to a point where it would feel like that guy is real, you just would not be sure if he was acting or keeping it real :)

Now this is a whole lot of data to stream. But we have 3gb/sec SDDs on the highest end, compared to 5400 rpm HDDs with less than 50mb/sec just two days ago; we have 64bit systems with tons of ram compared to 4GB max yesterday and we have 3D stacked memory on the GPUs, lots of it at very high bandwitdh. This is where Carmack magic should come in handy.

So if you are putting together a VR movie experience, like Story Studio is meant to do, and you are showing it to people on your own PC build, it could be this monster of a PC, that will be common in a year or two among hardcore VR fans and available to them this summer, and it could take advantage of knowing where the viewer is and knowing where everything is in relationship to it. Just like for movies, where first the storyboard is made, the experience can be first designed in engine, with simple 3D models and environments. When that story design phase is complete, assets could be replaced with those steps above depending on their interactivity, distance etc.

With a story that would tie in a helmet you have to keep at all times, an AR helmet, that keeps you alive in this hostile environment, the experience could claim that you are in the real world and these are real peole around you. Visually, it would be hard to distinguish that from really being there with such a helmet on your head. You would know that you have a helmet, there is no getting around that with the SDE and low res, but you would not know if you were there with a helmet on your head or just in VR. This game seems to have such a story (just guessing from the helmet shot and videos). When the displays in the helmets hit 16K per eye around 2018-2020, a story tie in for a helmet may no longer be needed as everything could look very real.

Actually I have an idea for an Oculus demo with Nate Mitchell in it. He is standing behind you, he tells you to put on the headset and you do. It is supposedly a demo for hand tracking, as a form of augmented reality with the Rift, replacing your hands in the video feed with robot hands. He says he is switching the cams on the headset on, you hear him to your right, next to you. You get the "video feed" you see him to your right. But he is simply not in the room anymore, in the real world. You hear him there with positional audio and see him there with 4D captures, your hands being rendered with handtracking and the environment being a laserscan to render, not a video. It should be possible to replicate the room as Oculus started to design their own demo booths instead of just showing the demos in wherever. Nate tells you to take off the helmet. He is not there. You put on the helmet again, he is still there on your right, but he also is walking in on your left, and there are two Nate Michells now. You take off the helmet, there is a TV in front of you, showing you how he left the second you put on the helmet. How more convincing can a VR demo get?

And that is the new Hollywood. A mix of game tech, on location laser scanning, scanning bodies for models, capturing performances for motion, capturing actors in 4D, and displaying in VR, with post effects, fog and noise tying everything together. We havent seen an example of this yet, the Kite & Lightning demo is probably as close as it gets to this on todays hardware. But that is about to change this summer because hardware is changing.

So come E3, either Oculus is going to be unleashing the beast, or it will drift further and further to mobile space while Valve unleashes the beast :)

And this could be the 4K version, released at the same time, or as and upgrade later. I'd hope for a single dp1.3 input but 2 dp1.2 would probably be the way to go as that would do the trick and be more compatible with stuff already out there and maybe even be cheaper to produce for a while.



Still used with a single, replaceable cable, combining the three.

Taking into account how important Palmer considers haptics to be, my guess for the final design would include this:



with maybe two types of options for gloves:
One cheap and light led glove, maybe with patterns on the outside facilitating tracking, like these:
https://twitter.com/ID_AA_Carmack/status/526905527230824448
And a second one, which is the same thing but with haptics.

With wristbands having IR cams facing both sides of the hand, never losing sight of the fingers, this should make pretty solid tracking possible. Again... working on both the PC and mobile lines.

"donkaradiablo" wrote:
I find the risks associated with this to be very similar to the ones for cannabis. One is emotional dependency and addiction by emotional dependency. The other one is how it can affect the perception of reality at a young age where the brain is still developing. I think we will see a lot of controversy centered around these two. I would advocate the legalization of cannabis but with an age limit, so it's hard for me to say VR would be safe for children. You get haptics in there, the father/mother who past away looks real, feels real and the emotional impact would be hard to master. That could also take in-app purchases to a whole new level.

On the other hand, this will open up so many possibilities for therapy. Guided VR sessions with pharmaceuticals. It might be a healthy way to say good bye to loved ones, under supervision from a pro. Or to overcome a bad memory, replacing it with a virtual one with repeated sessions. Honestly, that's gonna make todays therapy sessions look like a joke. On the flip side, brainwashing will get to a whole new level.

Very true. Therapy of the future is almost certain to involve VR at least for some approaches to treatment (it's already being used now).

It's also the basis of "the real Matrix". Morpheus gives a bunch of misfits drugs in the form of pills, hooks them up to the Rift, gives them trips with illusions of grandeur. Using manipulation through suggestion, cookies with drugs, shots to the arm and even drugs hooked to the whole body right after he shaves their heads "like a monkey, ready to be shot into space", conditions them to become terrorists ready to go blow up buildings and shoot people and that's okay because "Businessmen, teachers, lawyers, carpenters. The very minds of the people we are trying to save. But until we do, these people are still a part of that system and that makes them our enemy". Dresses them nice, takes them to clubs, makes them cool (all in VR). Trains them in VR, gives them a bunch more drugs and "lots of guns" and sends them to places to blow up. Of course he's just doing what "The Oracle" tells him to do, which makes him the perfect manipulator because he doesn't know that he is lying, he's on drugs too. And Zion is just another VR program they hook up these morons after they are done with them. (Would probably be possible with the right amount of drugs and subjects at the right age.)

When I think about how lines like "it's so powerful, it could be used by bad guys, so they are limiting export" were tossed around for Japanese consoles, how rumors like "they trained with this flight simulator" were tossed around after a worldwide tragedy, and how many controversies were centered around games and music videos, it is natural to expect more BS centered around VR than MJ. You've got parents concerned with what kids are watching on TV? Wait till it's in an alternate reality, surrounding the kid, something that the parent can't even see from the outside.

It doesn't even have to be crime of the century, the "Clockwork Orange" you can't look away, "Room 23 in Lost" that is a dream like state thanks to pharmaceuticals, "Dark City" style impanting fake memories, by repeating them this way a million times under the influence until the brain can't tell what is real... Could be as simple as getting a guy drunk, letting him wake up with his hands tied behind a chair and a VR device glued to his face using some simple local anesthesia, getting him to empty his bank accounts via his voice acticated smartwatch and fingerprint activated smartphone because he thinks his family is in danger. Or the subject in the same position, might think he/she is having intercourse with a person, while in reality it's just another person.

Some weird things are bound to happen with VR... But we may never hear of them because the victims may never figure out what really happened.

"donkaradiablo" wrote:
1. Rendering passes and parts of the moving world where the eye is more sensitive to temporal resolution, rendered at full speed, at half resolution
2. Parts of the 3D world where the eye is more sensitive to detail , rendered at half speed, full resolution, timewarped
3. Layered on top of each other with a fast pass of noise on Z data

If layering in the next SDK is as flexible as that, or is intended to get to be in the end, we can see some cool stuff.

It could also be cool to have a very low resolution flexible screen panels at twice the frame rate of the main display, located in the hmd around the lenses, used to provide ambient lighting beyond the displays fov. Could simply interpolate ambient color at 2x fps, creating another layer, in hardware.

Add to the list a fast pass, low res, low quality, high fov layer in the background with the current high quality low(er) fov layer on top in the center, for high fov PC headsets. The high quality image is rendered with more detail on the center and gets blurred around the edges anyway, so it creates a natural transition. If eye tracking is there, the center image can move accordingly. Could be partly done during timewarp.


______________________________________________


Also, wrote this in another topic, it wouldn't hurt to have it here too:

The adoption rate potential of mobile/portable VR is huge if you think about:

1. The way Wii got to people who weren't interested in gaming at all
2. The way mobile games have done a similar job
3. The adoption rate of the hardware for smartphones
4. The adoption rate of portable, compact stuff that you see on other people, like the Walkman, the iPod, Gameboy, Nintendo DS
5. How almost everybody walks into a mobile store at some point
6. How mobile service providers are eager to offer zero-down payment plans to keep you as a customer
7. How a platform with content delivery, app delivery, payment system, that comes with a portable device, is practical for the consumer, a cash cow for the platform owner
8. How a phone hardware manufacturer, who is also a display manufacturer, can sponsor content like 60 fps K-pop videos to promote it's TVs today, can sponsor content made for these headsets tomorrow, with some obvious intersection in between.
9. How bad Samsung wants to be "the next big thing", be seen as the "technology leader", like Apple, instead of "that company in the far east that you can make build stuff" and how they are willing to spend money on PR to push this aggressively.
10. How easy and practical and cheap it is to just pick up this futuristic looking thing for a portable personal theater type use, for educational programs in schools, for waiting rooms, for a bluetooth capable little box on you, streaming videos on the fly from your drone, for a "see for yourself" room next to changing cabinets in shops to give you an out of body experience to see if that dress makes your butt look fat... You get the picture.


You walk in to a shopping center, a mobile shop, a Walmart, you see this new thing Samsung has on display, between the phones and the consoles. It says "virtual reality", it says "try it". You put it on, it has magical Story Studio content, a virtual movie theater, 3D 360 degree K-pop videos, fun games, Youtube, Netflix, Hulu... It also says "zero down".

___________________________________________________________________


The problem with this approach being the main focus for Oculus:

1. You may have a genius like Carmack, solving your problems in software, providing tracking and lens correction and everything. None of that would matter when big companies come in, using brute force in hardware approach like LightHouse, or Wearility lenses, going large scale production, bringing the cost down for those, meaning all that time spent on trying to do things on the cheap has no value anymore.
2. Before Apple comes in, Google enters the mobile market with a standard of it's own, that works with all smartphones that follow it, not just a few.
3. Apple comes in to market. It comes in with Disney apps and Marvel games in 2017. Nobody would care about you anymore. Not even with the full marketing power of Samsung. Apple watch outselling all the smart watches combined to date in one day is a good example of this.
4. Not only that, but you have lost your cult following by switching to a platform that limits what you can offer, giving up on everything ypou have claimed were needed for VR, like 75-90fps and positional tracking, by saying "the consoles are not powerful enough for what we want to do" first and saying "you know what, people buy this, so it's not bad" for a mobile SOC powered product.

___________________________________________________________________


The solution "in an ideal world":

You merge your mobile and PC lines with a design

1. Allowing positional tracking on the portable side of things,
2. Allowing PC to be connected to the same device, using the same app/game/content platform, to run the same content at 90hz, at higher quality
3. Allowing a cutting edge PC powered flagship title to become your posterchild, that is also supported on mobile, but the screenshots and videos people see are from the PC version and hard to not be impressed.
4. Allowing you to keep beeing the company who pushes sales for cutting edge hardware like upcoming stacked memory GPUs, faster CPUs, brand name PCs, making every PR move they make mentioning VR a promotion of your product.
5. Allowing better hand tracking than anyone, possibly helped with two types of gloves:
One cheap and light led glove, maybe with patterns on the outside facilitating tracking, like these:
https://twitter.com/ID_AA_Carmack/status/526905527230824448
And a second one, which is the same thing but with haptics.

Again... working on both the PC and mobile lines.

___________________________________________________________________


So the advantage Oculus VR has is:

1. Being in both the mobile/portable and cutting edge hardware PC markets. The problem that comes with it is "not being laser sharp focused" and at least at the hardware side of things, that can stop being a problem by following a design that lets the two lines merge.
2. On the platform side of things, you have your mobile platform running and accepting payment worldwide now, and your PC platform will follow.
3. On the software side of things, the tools that can be used to create content, like Unity and UE, work for both mobile and PC. They just need a Carmack renderer for each platform, one that is mobile optimized, one that is PC optimized, and both using the Vulkan api that is supposed to be perfect for this job. Remember how many rendering paths Doom 3 had, or Quake 2 had? How it was not the renderer that took Carmack's time for id tech 5 (Rage), but other parts ot the engine? Considering how UE4 code is out there and builds with a different renderer are possible, this should be a no brainer.

___________________________________________________________________

This year has been hard on them, with pressure coming in to get this thing out the door fast from all sides: developers, partners, fans... But they have done a lot:

1. They have perfected the HMD design with GVR2, in terms of the straps used, weight distribution, fan to stop overheating and lens fog, a design to allow adjustment of focus. Also CB design with dual display panels, allowing them to use panels that come out of production for the Galaxy line, perfectly good for square shaped small panels but faulty beyond that.
2. They have perfected the platform with feedback from the community and all the time and effort it takes to have a paid app store worldwide.
3. They have perfected the design guidelines and integration with tools used for creating content.
4. They have some in-house made content and some third party made content to launch the platform with.


___________________________________________________________________

You make a great name for yourself and build a platform, thanks to that Carmack magic allowing you to do some things in software before the hardware that can be used to do them goes mainstream and down in price, you get the benefit of being the first to market. But for that to work, YOU HAVE TO BE FIRST TO MARKET!

All that needs to happen now is:

1. The demo of a flagship product in E3, with visuals that would get everyone hyped up, like the one posted in this message.
2. An announcement for the final design of the thing being surprisingly neat and exciting. And with an input solution that feels natural and is futureproof in terms of the gesture design it allows being portable to the following generations
3. And an announcement that takes place at the E3 event for limited availability during the summer, that allows developers to finish their content for the final design, the early adopters to be the guinea pigs for the final test run of that design.
4. To turn the hype train into a hype rocket, by reaching record sale numbers in days, the announcement states that this thing is available for ordering now at their website, for a limited edition shipping in summer, and preorders to be first in line for the high volume consumer version launch, happening Q4 2015/Q1 2016 and picking up speed in Q2 2016. So if you like what you see, you better order fast...

If you think about it, it's a limited launch that makes them "first to market" with a consumer version, and a mobile version, and a platform that works on both. And the high volume to market starts happening at the same time the mobile consumer product goes on sale and is promoted by Samsung.

An imagining of Half Life using scan based head models with advanced skin shaders for NPCs

(mock-up using HL2 Update and a photo)

Hoping it could be done by using something like Otoy's Lightstage:

(2009)

Probably with 4D captures of the face for scripted parts of the game.

"donkaradiablo" wrote:
When I think about how lines like "it's so powerful, it could be used by bad guys, so they are limiting export" were tossed around for Japanese consoles, how rumors like "they trained with this flight simulator" were tossed around after a worldwide tragedy, and how many controversies were centered around games and music videos, it is natural to expect more BS centered around VR than MJ. You've got parents concerned with what kids are watching on TV? Wait till it's in an alternate reality, surrounding the kid, something that the parent can't even see from the outside.

see:
http://www.gamesradar.com/sony-removed-suicide-option-project-morpheuss-heist-because-it-was-too-stressful/

"In the weeks ahead, we’ll be revealing the details around hardware, software, input, and many of our unannounced made-for-VR games and experiences coming to the Rift. Next week, we’ll share more of the technical specifications here on the Oculus blog."



E3...
A new IP with Carmack magic
Set to define the next step in first person games

Set on a land that feels magical
Like being inside a Disney/Pixar universe
You are inside an animated universe... after all

It has story studio made experiences scattered around
It feels like a dream
A lucid dream where you take control and you let go of it
And have experiences that almost feel spiritual

Or nostalgic... like going into an arcade game you played as a kid
And an adventure game... And an animated movie.

The trailer ends with a cutscene where a familiar voice is saying "feel the energy between your hands"
And there is an energyball/lighting/fireball forming between your "virtual hands"