With Nvidia Omniverse coming soon, will it be possible to have actual real-time full-body tracking soon with a “normal” webcam as they showed us in the demo? I'm really wondering
From what I can see (on Nvidia's Omniverse page) it wasn't Omniverse doing body tracking.
Omniverse was the platform that did the rendering. The actual body tracking was a separate project from Nvidia Research that used RTX cards to do pose estimation. That data was then sent to Omniverse to render.
Omniverse is basically a bridge between multiple programs intended for professional artists. From the announcement page: "For example an artist using Maya with a portal to Omniverse can collaborate with another artist using UE4 and both will see live updates of each others’ changes in their application."
(Not exactly efficient, it used 27% of my RTX2080TI just to track my skeleton)
The main problem with these single camera based body tracking systems:
- Lots of occlusion. It's very easy to block the view of body parts, which loses tracking.
- No inertial tracking. Normal tracking devices use accelerometers to help tracking when the cameras get occluded.
It can be an acceptable cheap solution though.
The best way to do full body tracking would be a mocap suit like a Perception Neuron. A bit expensive (around $1500 for the lower model), but that gives you 32 sensor inertial tracking. It would cost far more to have the same number of joints with Vive trackers.
@kojack : Do you think a similar program using two cameras could solve the occlusion issue? Like, imagine having external sensors, you place them in opposite angles (like the good ol CV1, or the Valve Lighthouses). It seems to me that it would be an obvious improvement, but no one tries it so maybe it wouldn't work that well for some reason.
I was wondering the other day: technically, the Quest is actually capable of body tracking. With the way the cameras are placed, your lower body is almost always in their FOV, right? That means Oculus would just have to make it recognize legs, and boom! Leg tracking! Of course you wouldn't be able to dance, it wouldn't be perfect body tracking, but it would still be a nice experiment. Hopefully that's Facebook's next objective for their avatars. EDIT : Basically this, but naturally, without any add-ons or markers on the floor : https://www.youtube.com/watch?v=m2Zu50vj5W4 (I forgot it existed, lol).
Far as I know your legs are not in the FOV at all times - thus they would need something like what is found inside the controllers. While possible to "guess: where the legs are then update when you kick/move to some level - it be a bit harder to do for normal use case. Controllers don't really cost much - maybe 25$ a ring so I am not sure why Oculus doesn't just sell a ring for tracking use cases.
"Nvidia Omniverse - Could it enable FULL body tracking?!"
Not sure I'd like that - there are body parts not meant to be tracked! :blush:
Valve Index & Oculus Rift CV1, Asus Strix OC RTX™ 3090, i9-10900K (5.3Ghz), 32GB 3200MHz, 8TB "Ask not what VR can do for you, but what you can do for VR"
