Error dialog "Cannot Purchase Additional Content" when trying to test IAP and/or subscriptions

We are calling ovr_IAP_LaunchCheckoutFlow(sku) with a valid sku. We are expecting it to either pass or fail depending on which test credit card we use but instead we get a system dialog saying this

"Cannot Purchase Additional Content"
"<our app name here> is required to purchase Monthly Subscription."
"OK"

Is something not configured correctly for our app or our IAP and Subscription SKUs? Our app id is 4784171281624823

We are using Unreal 4.25.4 Oculus branch. We think everything is configured correctly, we have this in DefaultEngine.ini

[OnlineSubsystem]
DefaultPlatformService=Oculus
bHasVoiceEnable=true

[OnlineSubsystemOculus]
bEnabled=true
OculusAppId=4784171281624823
MobileAppId=4784171281624823

We have tried both OculusAppId and MobileAppId because without MobileAppId we get errors saying it's not configured, we are on Quest only though think MobileAppId is for Rift.

The other possible issue we see is that when compiling a shipping build it gives us this error in the Output:

UATHelper: Packaging (Android (ASTC)): GetSignatureToken get_signature error: Missing entitlement for 4784171281624823 (1971031)
PackagingResults: Error: Missing entitlement for 4784171281624823 (1971031)
UATHelper: Packaging (Android (ASTC)): requestOafRoomUpdate response error: Must call get_signature first (1971051)
UATHelper: Packaging (Android (ASTC)): requestOafLaunchDetailsUpdate response error: Must call get_signature first (1971051)
PackagingResults: Error: Must call get_signature first (1971051)
PackagingResults: Error: Must call get_signature first (1971051)

But again that's during build, it's unclear if this affects IAP in an app that's been uploaded to the store. We are uploading to our ALPHA channel when trying to test all this.

Can someone point us in the right direction?

Thanks!