Streaming with passthrough enabled

I've been attempting to build an app that allows people to watch a stream of what I'm seeing. Everything is good until I stream with passthrough, it ends up being all black. I've been attempting to fix this for a couple days and just can't get it working. Is this something that's just not possible? I'm working with Unity and the passthrough works on the headset but when watching the stream it's just black. When I switch to VR it does work but the second passthrough is enabled, no dice.