Custom Hands shrink when adding colliders

So my job has insisted I learn game dev/vr dev for better or for worse and I'm having a pretty bad time of it, I do not get along well with this at all and the squeeze is on for "progress".

I've been trying to add colliders to hands (using controllers) so I can just make a demo where in objects get interacted with. The hands always just pass through things; or when I add a collider to them they start really small and grow back to the normal size when I press the trigger. At that point it works fine collisions work exactly as you'd hope, grabbing, etc, until I release the button and they snap back down to miniature size.

I'm on 2020.3.4f1 and using oculus integration. Anyone have any idea what's going on here?