Android MediaCodec decoding gets slower if tracking state is not normal

Dear all,

I'm working on decoding realtime video streaming on Quest.
A Unity VR application contains a native plugin which uses MediaCodec API for decoding HEVC streaming.
It works like a charm at 72 FPS, but strangely gets slower to about 40 FPS when tracking is lost or the guardian is turned off.
Specifically the output buffer available callback is called slower even though enqueue input buffers fast.

Is there anyone has an idea? Any kind of advice will be helpful :)