Quest 3 Build has Lag on Startup but Not Quest 2 Build

I've come across a very baffling issue where my VR game has performance issues on Quest 3 but runs perfectly smoothly on Quest 2. On Quest 3, the first 5 seconds of startup are unbearably laggy before it stabilizes. 

I was initially on Unity 2022.3.1f1 and my Oculus XR Plugin version was 4.0.0, which was supposedly the latest version in my package manager. I looked into the Oculus plugin patch notes, saw there was a 4.3.0 version, and realized that there wasn't even dedicated Quest 3 support for 4.0.0. I upgraded to both Unity 2022.3.16f1 and upgraded my Oculus XR Plugin version to 4.3.0 which helped reduce the dropped frames on startup for Quest 3, but it's still there. I think this has to be a plugin/unity issue because my Quest 2 build is working perfectly fine.