Colocation and Shared Spatial Anchors result in wrong frame of reference

Hello everyone!

I'm developing a networked experience using Unity 6 with Netcode for Gameobjects and the MetaXR SDK. I am trying to get Shared Spatial Anchors working using Colocation so that the players can see the same frame of reference in Mixed Reality.

I managed to get Colocation and Shared Spatial Anchor sharing to work but when I try to align my player rig using the Shared Spatial Anchor from the host there is a misalignment in the scene.
No obvious errors or warning are being thrown,I am using the code in the documentation as well as a video tutorial from "XR Dev Rob - Colocation with Meta’s Shared Spatial Anchors & the new Colocation Discovery API"  as my main point of reference for my own code.

Below is my alignment code

if (await unboundAnchor.LocalizeAsync())
{
   Debug.Log($"Anchor localized successfully, UUID: {unboundAnchor.Uuid}");
    var anchorGameObject = GameObject.Instantiate(_anchorPrefab);
    var spatialAnchor = anchorGameObject.GetOrAddComponent<OVRSpatialAnchor>();
    anchorGameObject.name = $"Anchor_{unboundAnchor.Uuid}";
    unboundAnchor.BindTo(spatialAnchor);
    _alignmentManager.AlignUserToAnchor(spatialAnchor);
    return;
}

With AlignUserToAnchor(OVRAnchor anchor

public void AlignUserToAnchor(OVRSpatialAnchor anchor)
{
    if (anchor == null || anchor.Localized == false)
    {
        Debug.LogError("Anchor is not localized yet.");
        return;
    }

    StartCoroutine(AlignmentCoroutine(anchor));
}

private IEnumerator AlignmentCoroutine(OVRSpatialAnchor anchor)
{
    var anchorTransform = anchor.transform;
    for (int alignmentCount = 2; alignmentCount > 0; alignmentCount--)
    {
        _cameraRigTransform.position = Vector3.zero;
        _cameraRigTransform.eulerAngles = Vector3.zero;

        yield return null;

        Vector3 offset = anchorTransform.InverseTransformPoint(Vector3.zero);
        Quaternion inverseYaw = Quaternion.Euler(0f, -anchorTransform.eulerAngles.y, 0f);

        _cameraRigTransform.position = offset;
        _cameraRigTransform.rotation = inverseYaw;
        Debug.Log($"Aligned camera rig position: {_cameraRigTransform.position}, rotation {_cameraRigTransform.eulerAngles}");

        yield return new WaitForEndOfFrame();

    }
    Debug.Log($"Alignment complete");
}
}

I am wondering if I am missing something that needs to be initialized to get the spatial data in order so that the Anchors can get localized correctly.
Or if it is something else that I am missing.

Thank you for your help!

Kind regards