XR Controller (Action-Based) not tracking When "Oculus" Plug-in Provider enabled

I need to enable the following "Oculus" Plug-in Provider in order to use the Avatars SDK and use Meta's avatars:

But I'm also trying to use the XR Interaction Toolkit because I find it better than the Oculus Interaction SDK and I want to make it easier to support other headsets in the future. I can get Device Based XR Controllers to work fine but Action-Based Controllers aren't tracked unless I use the OpenXR option above, but that causes the Meta Avatars to not work. It's recommended to use Action-Based so I don't want to put Device-Based Controllers in.

Does anyone know if it's possible and what steps to take to use XR Interaction Toolkit Action-Based Controllers and still have Meta Avatars work?