cancel
Showing results for 
Search instead for 
Did you mean: 

Nvidia Omniverse - Could it enable FULL body tracking?!

hesten_unam
Level 4
With Nvidia Omniverse coming soon, will it be possible to have actual
real-time full-body tracking soon with a “normal” webcam as they showed
us in the demo? I'm really wondering

6 REPLIES 6

kojack
Volunteer Moderator
Volunteer Moderator
From what I can see (on Nvidia's Omniverse page) it wasn't Omniverse doing body tracking.
Omniverse was the platform that did the rendering. The actual body tracking was a separate project from Nvidia Research that used RTX cards to do pose estimation. That data was then sent to Omniverse to render.

Omniverse is basically a bridge between multiple programs intended for professional artists. From the announcement page: "For example an artist using Maya with a portal to Omniverse can
collaborate with another artist using UE4 and both will see live updates
of each others’ changes in their application."

Author: Oculus Monitor,  Auto Oculus Touch,  Forum Dark Mode, Phantom Touch Remover,  X-Plane Fixer
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2

hesten_unam
Level 4
Oh, I see. I just got it wrong.
We can still dream I guess 😄 
I don't think this is that far away from consumer products, it's just a matter of time.

Thanks for your answer, I looked it up a little and didn't figure it out :smile:

kojack
Volunteer Moderator
Volunteer Moderator
Well, there's still a chance Nvidia will release the pose estimation software.

There's some existing ones that do the same thing. For example PoseNet. Here's a demo that runs in a browser if you have a webcam: https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html
(Not exactly efficient, it used 27% of my RTX2080TI just to track my skeleton)

The main problem with these single camera based body tracking systems:
- Lots of occlusion. It's very easy to block the view of body parts, which loses tracking.
- No inertial tracking. Normal tracking devices use accelerometers to help tracking when the cameras get occluded.

It can be an acceptable cheap solution though.

The best way to do full body tracking would be a mocap suit like a Perception Neuron. A bit expensive (around $1500 for the lower model), but that gives you 32 sensor inertial tracking. It would cost far more to have the same number of joints with Vive trackers.
Author: Oculus Monitor,  Auto Oculus Touch,  Forum Dark Mode, Phantom Touch Remover,  X-Plane Fixer
Hardware: Threadripper 1950x, MSI Gaming Trio 2080TI, Asrock X399 Taich
Headsets: Wrap 1200VR, DK1, DK2, CV1, Rift-S, GearVR, Go, Quest, Quest 2, Reverb G2

Anonymous
Not applicable
@kojack : Do you think a similar program using two cameras could solve the occlusion issue? Like, imagine having external sensors, you place them in opposite angles (like the good ol CV1, or the Valve Lighthouses). It seems to me that it would be an obvious improvement, but no one tries it so maybe it wouldn't work that well for some reason.

I was wondering the other day: technically, the Quest is actually capable of body tracking. With the way the cameras are placed, your lower body is almost always in their FOV, right? That means Oculus would just have to make it recognize legs, and boom! Leg tracking! Of course you wouldn't be able to dance, it wouldn't be perfect body tracking, but it would still be a nice experiment. Hopefully that's Facebook's next objective for their avatars.
EDIT : Basically this, but naturally, without any add-ons or markers on the floor : https://www.youtube.com/watch?v=m2Zu50vj5W4 (I forgot it existed, lol).

Anonymous
Not applicable
Far as I know your legs are not in the FOV at all times - thus they would need something like what is found inside the controllers. While possible to "guess: where the legs are then update when you kick/move to some level - it be a bit harder to do for normal use case. Controllers don't really cost much - maybe 25$ a ring so I am not sure why Oculus doesn't just sell a ring for tracking use cases. 

RuneSR2
Level 15

"Nvidia Omniverse - Could it enable FULL body tracking?!"


Not sure I'd like that - there are body parts not meant to be tracked! :blush: 

Valve Index & Oculus Rift CV1, Asus Strix OC RTX™ 3090, i9-10900K (5.3Ghz), 32GB 3200MHz, 8TB
"Ask not what VR can do for you, but what you can do for VR"