Custom Models in Unity for Controllers?

Try as I might... I cannot figure this out exactly.

What I want to know is... if you're starting from a CORE VR project that implements the main XR rig, can I not just change the mesh to a custom 3d model and it show my 3d model instead of the normal controllers?

I've tried various different ways... and I cannot get it to work. It keeps showing only the original models Unity started with.

What am I doing wrong? I just want to use my custom 3d mesh model as left and right floating in game controllers in the game.

Thank you for your time.