Mirror, Mirror on the Wall, Which One Has the Best (or Worst) Tracking of Them All?

[RuneSR2]

Investigators behind a new scientific study have measured tracking accuracy using different hmds and controllers, so this isn't any autodidactic YouTuber expressing a subjective opinion. Instead - according to RoadToVR's translation of the original German paper - investigators did this:

As first reported by German publication MIXED (German), VDC Fellbach measured the tracking systems on Oculus Rift S, Oculus Quest, Windows Mixed Reality, Valve’s SteamVR (aka Lighthouse) tracking, and HTC Vive Cosmos. Ostensibly as a sort of control, the study also included the industrial-grade tracking cameras from ART, which routinely outperformed even the rock-solid external SteamVR laser-based tracking system.

The objective was to quantify each system’s input accuracy, measured in millimeters, which was done by attaching each controller to a robot arm with an adaptor. A 3D-printed head model was used to hold the target headset while the robot arm moved the VR controllers 50 times between two points on the X-axis, set 500 millimeters (~20 inches) apart, or about an arm’s distance away.

Image courtesy VDC Fellbach, translated by Road to VR

The robotic arm, which was connected to the laptop via LAN, was controlled by an in-house software using a modified variant of OpenVR. This allowed the testers to not only reliably know the actual position of the controller, but also the reported position of the controller, capturing data in the X, Y and Z-axis during each headset’s trial.

Source: https://www.roadtovr.com/htc-vive-cosmos-accuracy-test-controller/

Results - lower scores are best (hint: Cosmos did not win 😉





I've attached the original German paper to this post.

Not hard to find the most precise tracking, but what about second and third places? Now I haven't read the full German paper, not sure I will, but each hmd has 6 values - and if we take the averages for every hmd, we get:

1. Index = 0.130
2. Quest = 0.502
3. Rift-S = 0.643
4. WMR = 0.916
5. Cosmos = 2.577

Maybe pretty close to real-world experiences? 

Sad not to have CV1 included, maybe next time - to really see what Oculus sacrificed for inside-out... 

[Anonymous]

Very cool, am surprised that the Quest has better tracking than the Rift S in some cases given the extra camera that the Rift S has.

[pyroth309]

Seems about right, that's pretty damn good for all of them, except the cosmos anyway lol and Z axis on that position 2 test for WMR. The rest they're close enough that you'd never perceive a difference. Wonder what kind of motion they were doing on that 2nd test to get that bad result. First Z test was actually better than Rift and Quest. I tried to look through the document but I can't read german and don't feel like translating it all.

[pyroth309]

I think I figured it out, Position 2 was 500mm away and ran the same tests it seems.  So looks like WMR accuracy has more problems at distance at least on Z axis anyway.

I've used all of these systems tested except Rift-S and cosmos as well as the CV1 which isn't represented. Hands down Lighthouse felt the best to me when I used it when Index came out. Not only in accuracy but in volume. No matter where I had my hands the tracking was rock solid. But I only used it for 2 weeks.

I've used WMR for over a year and never ran into a situation where I felt like accuracy was a problem. Tracking volume on the other hand....

[RuneSR2]

pyroth309 said:

I think I figured it out, Position 2 was 500mm away and ran the same tests it seems.  So looks like WMR accuracy has more problems at distance at least on Z axis anyway.

I've used all of these systems tested except Rift-S and cosmos as well as the CV1 which isn't represented. Hands down Lighthouse felt the best to me when I used it when Index came out. Not only in accuracy but in volume. No matter where I had my hands the tracking was rock solid. But I only used it for 2 weeks.

I've used WMR for over a year and never ran into a situation where I felt like accuracy was a problem. Tracking volume on the other hand....

I guess these results just show why base station tracking is referred to as the "gold standard" - at least we got the numbers, which Valve, HTC, Oculus etc probably have had for ages.
I'd have loved to see CV1 tracking, I don't feel it's much worse than the base stations, but surely I might not consciously register a few extra milliseconds here and there... 

[RuneSR2]

snowdog said:

Very cool, am surprised that the Quest has better tracking than the Rift S in some cases given the extra camera that the Rift S has.

This one did seem somewhat odd to me too - then again, if I should guess then Oculus engineers may have prioritized Quest tracking more than Rift-S tracking...  

[RuneSR2]

PS. Would also be very interesting to see how G2 performs - the current WMR results may not represent G2 (or maybe they do - but G2 has more cameras that might improve precision). 

[DaftnDirect]

So those results are average deviation in mm? I haven't read the full thing so sorry if I've missed something.

I'm with pyroth in thinking all are very good. A fraction of a mm (if that's what it is) wouldn't be perceivable would it? unless you're interacting with something in the real world that's either very small (a mm across maybe) or requires very precise placement tolerances. I guess ART caters for industry that needs that level of accuracy.

If you're interacting with something in a virtual world, your virtual self wouldn't ever be aware of being even 5 or 10mm out of place as your virtual hands would be in whatever position are represented... involuntary movements caused by inaccuracy would be bad though and occlusion of course.

Hand tracking, as we have with the beta implementation on Quest is another thing altogether I think. Having tested VRtuos where you use the app to play or lean the piano using a real keyboard, accuracy is everything and by my reckoning the tips of your fingers can be 25mm out of place no matter how good you calibrate which is coincidentally just enough to have you play an adjacent key to the one you meant to!

Very good test though.

Edit: latency is the other big deal.

[pyroth309]

DaftnDirect said:

So those results are average deviation in mm? I haven't read the full thing so sorry if I've missed something.

I'm with pyroth in thinking all are very good. A fraction of a mm (if that's what it is) wouldn't be perceivable would it? unless you're interacting with something in the real world that's either very small (a mm across maybe) or requires very precise placement tolerances. I guess ART caters for industry that needs that level of accuracy.

If you're interacting with something in a virtual world, your virtual self wouldn't ever be aware of being even 5 or 10mm out of place as your virtual hands would be in whatever position are represented... involuntary movements caused by inaccuracy would be bad though and occlusion of course.

Hand tracking, as we have with the beta implementation on Quest is another thing altogether I think. Having tested VRtuos where you use the app to play or lean the piano using a real keyboard, accuracy is everything and by my reckoning the tips of your fingers can be 25mm out of place no matter how good you calibrate which is coincidentally just enough to have you play an adjacent key to the one you meant to!

Very good test though.

Yea it's deviation in mm and most are a fraction of 1mm in difference. Even the cosmos at 3.2 mm (which for us American types is about 1/8th of an inch) is still pretty accurate.

[RuneSR2]

Yes, and results from Cosmos sometimes were more than 5 mm off and easy to detect consciously.

I wonder how much tracking accuracy is impacted when you got low fps or am doing very fast movements. I don't think the investigation tried to check tracking accuracy like 360 degrees around a fixed hmd, which may have provided much greater differences and "blind spots". 

The original article has many more results, I might read it tomorrow.