Is it possible to design a game that uses Controller and Hand Tracking at the same time?

Currently hand tracking is used as a substitute of controllers and that it won't enable unless controllers sit idle for a few seconds, thus they cannot be used simultaneously to have controller and hand each govern a different set of commands.
Is this a hardware limitation or hard-coded restriction that cannot be worked around?

Say for example, if I want to swing melee weapons around in a RPG game with controllers for better tracking, but wants to be able to cast spells with hand gestures by "dropping" (using wrist band) my controller and quickly switch to hands to draw magical circles in the air. Once done I would pick up the dangling controller quickly and continue with melee, all of which makes for a seamless gameplay without ever opening an in-game menu to switch between melee and magic mode?