Tracked hand pushing away menu (canvas) attached to other hand

I am keeping my left hand almost completely still here in the video demo below, but something pushes the left hand or the menu away when I get the right hand near the other hand/ menu (hard to tell if tracking issue due to other hand being near left hand vs hand interacting with menu or menu item colliders or rigibodies.

Video of error in demo

Video of unity project setup

I have an empty gameobject called 'menu-hand_attatcher' with this script on:

using System.Collections;
using System.Linq;
using UnityEngine;
using Oculus.Interaction;   // for OVRHand & OVRSkeleton

public class AttachMenuToHand : MonoBehaviour
{
    [Tooltip("The disabled Canvas or prefab you want to spawn")]
    public GameObject menuPrefab;  // assign your menu_0 prefab here

    [Tooltip("Which bone/joint to anchor the menu to")]
    public OVRSkeleton.BoneId anchorJoint = OVRSkeleton.BoneId.Hand_WristRoot;

    [Tooltip("Offset from that joint in metres")]
    public Vector3 localOffset = new Vector3(0f, 0.05f, 0.06f);

    private OVRHand hand;     // we’ll find this automatically
    private bool attached;

    private void Awake()
    {
        // Auto-find the first active OVRHand in the scene (left or right)
        hand = FindFirstObjectByType<OVRHand>();
    }

    private IEnumerator Start()
    {
        if (hand == null)
        {
            Debug.LogError("❌ No OVRHand component found anywhere in the scene!", this);
            yield break;
        }

        // Grab the OVRSkeleton attached to that hand
        OVRSkeleton skel = hand.GetComponent<OVRSkeleton>();
        if (skel == null)
        {
            Debug.LogError("❌ OVRHand has no OVRSkeleton component!", this);
            yield break;
        }

        // Wait until the system has tracked the hand & built out all bones
        while (!hand.IsTracked || skel.Bones.Count == 0)
        {
            yield return null;
        }

        // Find the specific bone (wrist, index tip, etc.) by BoneId
        OVRBone targetBone = skel.Bones
            .FirstOrDefault(b => b.Id == anchorJoint);

        if (targetBone == null)
        {
            Debug.LogError($"❌ Couldn't find bone {anchorJoint} on the skeleton!", this);
            yield break;
        }

        Transform jointTransform = targetBone.Transform;

        // Spawn or enable the menu prefab
        GameObject menuInstance = menuPrefab.activeSelf 
            ? menuPrefab 
            : Instantiate(menuPrefab);

        menuInstance.SetActive(true);
        menuInstance.transform.SetParent(jointTransform, worldPositionStays: false);
        menuInstance.transform.localPosition = localOffset;
        menuInstance.transform.localRotation = Quaternion.identity;

        attached = true;
    }

    private void LateUpdate()
    {
        if (attached && Camera.main != null)
        {
            // Keep the menu facing the user’s camera
            transform.LookAt(Camera.main.transform.position, Vector3.up);
        }
    }
}

Anchor joint is set to XR Hand_Start

Items in menus have collider + rigidbody   (no gravity) to allow acting as trigger

It's not so clear in the video but it appears going near the colliders for the buttons freaks it out and flips the menu around, but so does just crossing the right hand over the left (menu anchored to left wrist), or getting it near the canvas. So hard to tell what its interacting with.

1. What can I try using this method I'm already using? (I'm very new to Unity and coding so redoing things or doing them a different way could take me a lot of time and I don't have much)

2. If I were to - is there a better way to do this menu/canvas and hand attachment?
Here's my script of my menus

using UnityEngine;
using UnityEngine.UI;

public class MenuNavigator : MonoBehaviour
{
    [Tooltip("Prefab of the submenu to open when this button is poked")]
    public GameObject submenuPrefab;

    // debounce flags
    private bool _hasActivated = false;
    private bool _ignoreNextTrigger = true;

    private Transform _canvasRoot;

    void Awake()
    {
        // cache the Canvas transform so we know where our menus live
        Canvas c = GetComponentInParent<Canvas>();
        if (c != null)
            _canvasRoot = c.transform;
        else
            Debug.LogError("[MenuNavigator] No Canvas found!", this);
    }

    void OnEnable()
    {
        // every time this menu (and its buttons) become active:
        _ignoreNextTrigger = true;
        _hasActivated = false;
    }

    void OnTriggerEnter(Collider other)
    {
        if (_ignoreNextTrigger) return;   // ignore the initial overlap
        if (_hasActivated) return;   // and only fire once per exit/enter

        // 1) Spawn the next submenu prefab under the Canvas
        if (submenuPrefab != null && _canvasRoot != null)
        {
            var newMenu = Instantiate(submenuPrefab, _canvasRoot);
            newMenu.transform.localPosition = Vector3.zero;
            newMenu.transform.localRotation = Quaternion.identity;
            newMenu.transform.localScale = Vector3.one;
        }
        else
        {
            Debug.LogWarning("[MenuNavigator] Missing submenuPrefab or Canvas!", this);
        }

        // 2) Find *your* menu panel: the direct child of the Canvas
        Transform panel = transform;
        while (panel != null && panel.parent != _canvasRoot)
        {
            panel = panel.parent;
        }

        if (panel != null)
        {
            Destroy(panel.gameObject);
        }
        else
        {
            Debug.LogWarning("[MenuNavigator] Couldn't find menu panel root!", this);
        }

        _hasActivated = true;
    }

    void OnTriggerExit(Collider other)
    {
        // 1st exit after spawn turn off ignore, subsequent exits clear the activated flag:
        if (_ignoreNextTrigger)
            _ignoreNextTrigger = false;
        else
            _hasActivated = false;
    }
}

Many thanks :)