I found a solution, it turns out the app was running on compatibility mode by default, and it was still treating the device as a Quest 1. I fixed it by setting minSdkVersion and targetSdkVersion to 32 (in both build.gradle and AndroidManifest.xml), and also to add the following to inside the application tag inside AndroidManifest.xml
<meta-data
android:name="com.oculus.intent.category.VR"
android:value="vr_only"/>
<meta-data
android:name="com.oculus.supportedDevices"
android:value="quest2|questpro|quest3"/>
My app is targeting Quest 2 as a minimum, so the solution might need to be tweaked if you still need to target Quest 1. More info can be found here:
https://developers.meta.com/horizon/documentation/native/android/mobile-native-manifest/