Since no one gave any solutions yet I'll add mine.
My solution was to use Premultiplied Alpha instead of typical Alpha Blend. I believe since Alpha Blend applies itself to the source image it overwrites the pixel's alpha. If that's right, the Passthrough then might think that these pixels are completely transparent.
Since Premultiplied alpha just adds itself to the image while checking the source on the side, the only thing it can really do is "add" more than 1 to the alpha which doesn't happen.
This is all conjecture though to be completely honest. The important part is that it works.
This is a fix for conventional blending only though. I've been using Alpha to coverage for a bunch of my shaders and I had to completely trash that in favor of premultiplied alpha with another depth pass on the side which is a real bummer. Meta should still find a fix for this so that Alpha to coverage is still usable in mixed reality.
One thing that' I'd love to have as a feature is to be able to just input the passthrough into the clear flag of the camera, since that would give all the control I personally would ever need over it.
