Credentials not passing entitlement check?

I've logged in successfully with my credentials in the platform settings, it always fails the entitlement check.

It works fine in a build on the quest, so I know the app ID is fine.

I've tried my own credentials (which worked for my last game), and test credentials created for the app.

I was using 2021.3.9 with an older version of the oculus integration and openXR, I have now upgraded to 2021.3.23 and updated all plugins (oculus integration 51.0), and still have the same issue.

I need to test the leader board so I really need this to work. Any ideas?