Soooo. Am i missing something or is this not illegal?
I noticed the issue cause my collider capsules attached to hand tracked bones weren't updating position/rotation with rigidbody.MovePosition/rigidbody.MoveRotation.
Removing it fixed my issue, although the physics update in other projects worked fine with the implementation.