Nvidia Omniverse - Could it enable FULL body tracking?!

[hesten_unam]

With Nvidia Omniverse coming soon, will it be possible to have actual
real-time full-body tracking soon with a “normal” webcam as they showed
us in the demo? I'm really wondering

[kojack]

From what I can see (on Nvidia's Omniverse page) it wasn't Omniverse doing body tracking.

Omniverse was the platform that did the rendering. The actual body tracking was a separate project from Nvidia Research that used RTX cards to do pose estimation. That data was then sent to Omniverse to render.

Omniverse is basically a bridge between multiple programs intended for professional artists. From the announcement page: "For example an artist using Maya with a portal to Omniverse can
collaborate with another artist using UE4 and both will see live updates
of each others’ changes in their application."

[hesten_unam]

Oh, I see. I just got it wrong.
We can still dream I guess 😄 
I don't think this is that far away from consumer products, it's just a matter of time.

Thanks for your answer, I looked it up a little and didn't figure it out :smile:

[kojack]

Well, there's still a chance Nvidia will release the pose estimation software.

There's some existing ones that do the same thing. For example PoseNet. Here's a demo that runs in a browser if you have a webcam: https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html

(Not exactly efficient, it used 27% of my RTX2080TI just to track my skeleton)

The main problem with these single camera based body tracking systems:

- Lots of occlusion. It's very easy to block the view of body parts, which loses tracking.

- No inertial tracking. Normal tracking devices use accelerometers to help tracking when the cameras get occluded.

It can be an acceptable cheap solution though.

The best way to do full body tracking would be a mocap suit like a Perception Neuron. A bit expensive (around $1500 for the lower model), but that gives you 32 sensor inertial tracking. It would cost far more to have the same number of joints with Vive trackers.

[Anonymous]

@kojack : Do you think a similar program using two cameras could solve the occlusion issue? Like, imagine having external sensors, you place them in opposite angles (like the good ol CV1, or the Valve Lighthouses). It seems to me that it would be an obvious improvement, but no one tries it so maybe it wouldn't work that well for some reason.

I was wondering the other day: technically, the Quest is actually capable of body tracking. With the way the cameras are placed, your lower body is almost always in their FOV, right? That means Oculus would just have to make it recognize legs, and boom! Leg tracking! Of course you wouldn't be able to dance, it wouldn't be perfect body tracking, but it would still be a nice experiment. Hopefully that's Facebook's next objective for their avatars.
EDIT : Basically this, but naturally, without any add-ons or markers on the floor : https://www.youtube.com/watch?v=m2Zu50vj5W4 (I forgot it existed, lol).

[Anonymous]

Far as I know your legs are not in the FOV at all times - thus they would need something like what is found inside the controllers. While possible to "guess: where the legs are then update when you kick/move to some level - it be a bit harder to do for normal use case. Controllers don't really cost much - maybe 25$ a ring so I am not sure why Oculus doesn't just sell a ring for tracking use cases. 

[RuneSR2]

"Nvidia Omniverse - Could it enable FULL body tracking?!"

Not sure I'd like that - there are body parts not meant to be tracked! :blush: