ISDK hand tracking issues

I posted this on reddit already. I am having issues implementing hand tracking with Meta XR Plugin and ISDK in UE5.5. 

Have tried both the ISDK sample project and setting things up from scratch according to the docs. I did it on two different PCs. I am using Virtual Desktop on my Quest Pro. Handtracking inside Horizon Os and VD works perfectly fine.

Even when I have no controller in my hand it would show the controller setup. I am able to grab stuff with that crooked hand but pinching does not seem to work, however, I can access the menus by rotating my hand and tabbing. If I set hand tracking to "Hands only", it would neither show hands nor controllers but I can still access the menu using the mentioned gesture.

Any idea what is going on here?