Mirror, Mirror on the Wall, Which One Has the Best (or Worst) Tracking of Them All?

[RuneSR2]

Investigators behind a new scientific study have measured tracking accuracy using different hmds and controllers, so this isn't any autodidactic YouTuber expressing a subjective opinion. Instead - according to RoadToVR's translation of the original German paper - investigators did this:

As first reported by German publication MIXED (German), VDC Fellbach measured the tracking systems on Oculus Rift S, Oculus Quest, Windows Mixed Reality, Valve’s SteamVR (aka Lighthouse) tracking, and HTC Vive Cosmos. Ostensibly as a sort of control, the study also included the industrial-grade tracking cameras from ART, which routinely outperformed even the rock-solid external SteamVR laser-based tracking system.

The objective was to quantify each system’s input accuracy, measured in millimeters, which was done by attaching each controller to a robot arm with an adaptor. A 3D-printed head model was used to hold the target headset while the robot arm moved the VR controllers 50 times between two points on the X-axis, set 500 millimeters (~20 inches) apart, or about an arm’s distance away.

Image courtesy VDC Fellbach, translated by Road to VR

The robotic arm, which was connected to the laptop via LAN, was controlled by an in-house software using a modified variant of OpenVR. This allowed the testers to not only reliably know the actual position of the controller, but also the reported position of the controller, capturing data in the X, Y and Z-axis during each headset’s trial.

Source: https://www.roadtovr.com/htc-vive-cosmos-accuracy-test-controller/

Results - lower scores are best (hint: Cosmos did not win 😉





I've attached the original German paper to this post.

Not hard to find the most precise tracking, but what about second and third places? Now I haven't read the full German paper, not sure I will, but each hmd has 6 values - and if we take the averages for every hmd, we get:

1. Index = 0.130
2. Quest = 0.502
3. Rift-S = 0.643
4. WMR = 0.916
5. Cosmos = 2.577

Maybe pretty close to real-world experiences? 

Sad not to have CV1 included, maybe next time - to really see what Oculus sacrificed for inside-out... 

[pyroth309]

RuneSR2 said:

Yes, and results from Cosmos sometimes were more than 5 mm off and easy to detect consciously.

I wonder how much tracking accuracy is impacted when you got low fps or am doing very fast movements. I don't think the investigation tried to check tracking accuracy like 360 degrees around a fixed hmd, which may have provided much greater differences and "blind spots". 

The original article has many more results, I might read it tomorrow.

Ouch. They should have done PSVR for the laughs and to give Cosmos some company. Of course it's kind of in a similar issue as WMR. Tracking isn't bad if you get the lighting right and stay in the camera. My aim controller always loses orientation though. I've heard people claim the tracking feels on par with Light house if you use a green hue light.

[RuneSR2]

I guess what we see from the results are the best possibly accuracy when you don't move the controllers.
I would think more pronounced differences might occur when we start swinging the controllers, lol - also depending on how often tracking/position gets updated - but fast movement precision might be harder to measure. Then again, maybe they did cover that in the article, I don't know.

[SkScotchegg]

My first post back on these forums in maybe a year!! I was on a secret mission to find the best VR headset in the world! lol

I think I finally found it! HP Reverb G2! This headset sounds amazing.

Anyways, I just wanted to add to this thread that you can use The Index tracking and knuckle controllers with the HP G2.

I really want that setup! The best headset in the world with the best tracking and controllers! I can't wait but I think it will cost over £1000 in the UK which is expensive but I guess it's got to be done!

Nice thread anyways, it's a shame about Rift S tracking, CV1 was amazing.

I've been out of the VR loop for a long time. Is there any news of CV2 yet?

[pyroth309]

SkScotchegg said:

My first post back on these forums in maybe a year!! I was on a secret mission to find the best VR headset in the world! lol

I think I finally found it! HP Reverb G2! This headset sounds amazing.

Anyways, I just wanted to add to this thread that you can use The Index tracking and knuckle controllers with the HP G2.

I really want that setup! The best headset in the world with the best tracking and controllers! I can't wait but I think it will cost over £1000 in the UK which is expensive but I guess it's got to be done!

Nice thread anyways, it's a shame about Rift S tracking, CV1 was amazing.

I've been out of the VR loop for a long time. Is there any news of CV2 yet?

Just keep in mind you need some Steam Dongles either off the steam controller or off the Vive trackers to achieve that and they're getting harder to find. Also Valve isn't selling base stations any time soon, most likely just to harm HTC. 😄

[kojack]

Their terminology is backwards.

Lighthouse is inside out for controllers and headset (people still don't understand this, they think external box means outside in)

Quest, Rift-S, WMR and Cosmos are outside in for controllers and inside out for headset.

Since the test is measuring just controller tracking, they should be referring to the tracking system used by the controllers, not the headset.

Technically the IMUs in the controllers are inside out, but position tracking is still primarily affected by the outside in camera tracking of light markers on the controllers.

Anyway, terminology fail aside, what they should also test is orientation tracking. Being off by 1mm in position is a very minor issue in 99% of cases, but being off by 1 degree of rotation is going to have a big effect on aim in a game with long range weapons.

Also, what is the stability of the headset tracking? They have the headset mounted on a head, but remember for Rift-S, Quest, WMR and Cosmos the position of controllers are relative to the headset. If the headset position or orientation is noisy, that will add to the error of the controller tracking.

The article seems to suggest they used OpenVR for everything. It would be interesting to test with native oculus sdk too, to see if OpenVR is affecting the accuracy (perhaps they do smoothing or a coordinate system transformation).

Hmm, actually that last one I should be able to do myself, I don't need an industrial robot arm to compare coords between two apis.

[RuneSR2]

After reading most of the original article, I'm thinking RoadToVR picked some of the nicest results. The investigators measured maximum errors (worst case scenarios or close), average values and the standard deviations (latter shown in my original post in this thread). 

When I bought my 3rd CV1 sensor, it wasn't because 2 sensors on average were bad - they were great. It was due to the worst case scenarios - the few extreme cases in Lone Echo where tracking was bloody awful (excuse my French) - that I bought my 3rd sensor. 

Therefore I might find these results more interesting (Y Axis illustrates millimeters, lower score indicates better precision):



Now we're no longer talking about a few millimeters - but more than a centimeter for the Cosmos. Close to 0.5 centimers for Rift-S. 

Also the results aren't based on just a few measurements, but 50 - like shown below - and if you can see a colored bar it's not necessarily a good thing 😉



I think most of us find it easy to see violet and yellow, but red and green are harder...

Of course the above results do not measure frequency and volume - I was struck by a few comments to the RoadToVR article, thus the above results may just be considered the tip of the iceberg...

"TBH, the reason I play beatsaber on my Index is because of the tracking volume. I find that when I get really into things, my hands leave the tracking volume of the oculus quest quite frequently. It's much less about tracking quality and more about volume and consistency."


"I own the Index and the Quest. There is a huge huge different between the two during Beat Saber and other games. Index is so much more accurate when moving fast and slow. Especially when moving fast because the Quest will lose tracking frequently for short moments."

https://www.roadtovr.com/htc-vive-cosmos-accuracy-test-controller/

[pyroth309]

RuneSR2 said:

I think most of us find it easy to see violet and yellow, but red and green are harder...

Of course the above results do not measure frequency and volume - I was struck by a few comments to the RoadToVR article, thus the above results may just be considered the tip of the iceberg...

"TBH, the reason I play beatsaber on my Index is because of the tracking volume. I find that when I get really into things, my hands leave the tracking volume of the oculus quest quite frequently. It's much less about tracking quality and more about volume and consistency."


"I own the Index and the Quest. There is a huge huge different between the two during Beat Saber and other games. Index is so much more accurate when moving fast and slow. Especially when moving fast because the Quest will lose tracking frequently for short moments."

https://www.roadtovr.com/htc-vive-cosmos-accuracy-test-controller/

I agree with the comments. As I mentioned up above a few posts back, tracking volume is a huge piece of the puzzle especially in games like Beat Saber. When you get to Expert+ and you're trying to swing super fast consistency is very important. Even a centimeter isn't going to affect your score that much as the center register on the boxes is pretty lenient. What is very much more important though is not losing tracking which a large tracking volume provides. You don't get your controllers wigging out while the game is trying to recalculate where the hell you are on Index and CV1 because they have such large tracking volume.

I've played a lot of beat saber on both Quest and WMR and while each have some pros and cons I'm just as proficient on either system and I can complete and fail on the same maps with a few notable exceptions. There's a few maps I've tried that required a lot of high and swinging wide stuff that I run into instances where one of my controllers ends up out of the camera for more than 2 seconds where the Quest upward cameras can still see. Still, there's times on the Quest for some reason that even though I'm where the camera should be able to see it, it hiccups and loses tracking. It's super annoying in a game like Beat saber. 

With Index, I was able to just jump in and swing any kind of way I wanted and it was mirrored in game. CV1 was the same way unless I got near the corner where i didn't have a camera anyway.

That all being said, accuracy can be important in certain situations, especially if you're using muscle memory to make a snap shot or something. You would expect the gun or whatever in game to be where it needs to be.