Note to all developers looking to use the overlay layer to render 360 content. There are issues within the Oculus overlay implementation which makes 360 content not a viable option.
The overlay layer is rendering straight to the eyebuffer(screen) and needs special care to do so. The edges of an overlay layer tend to have a thin dark outline which is usually not an issue with a quad or cylinder as they can be easily masked by darker backgrounds.
Now to render 360 content with the overlay layer, you need to use a equirect layer. The equirect overaly shapes render 360 video/images clear and sharp however there is a massive issue making the equirect option unusable. The edges of the overlay layer still produce a small black line which happens to be on the back side of the sphere. Looking forward is just fine however once you turn around, the black line is always at the back of the video. The line is very thin but extremely noticeable. A strange aspect of the black line is that it is extremely tiny at the poles of the sphere but it is its thickest at the equator, though it is still very thin.
I've reproduced this with 4k mono video and stereo video and 360 images. It happens with AVPro, Unity's video player and still image textures. I've primarily tested this with the latest oculus utilities 12/1.44 and Unity 2018.4.9 and 2019.2.3. In my research I also reproduced it on older version of oculus utilities which produced similar results with early versions producing an even thicker line.
I am not the only developer who has come across this issue. I would post links to these discussions however, I "have to be around for a little while longer before you can post links". They can be found by searching "overlay" in the forum. In addition, John Carmack replied to a twitter thread about this on July 10th, of 2018, however his responses did address the real issue.
In addition to these posts, I have privately spoken to other developers who have had the same exact issue and ended up having to ditch the overlay layer all together to properly render 360 content. I was met with the same fate as my counterparts. I have concluded that it is not possible to properly render 360 content on the oculus overlay layer.
Other interesting tidbits i've found about the equirect overlay layers: Oculus offers an example scene with the equirect overlay layer but it is only a 180 video. The entire back half of the sphere is black so the black line not visible. I inserted my own 360 content into this demo scene and switched its custom video player script to 360 from 180 and of course, the black line was there. Oculus does not actually give a demo of the 360 content rendering on the overlay layer. I have personally never seen a single instance of any 360 content rendering on the overlay layer without the black line(within unity). In addition oculus does not actually provide an example of 360 content rendering on the overlay layer, though they have a demo for the much less utilized 180 video. I believe oculus did not deliver any demo scenes with 360 content because they know it doesn't work.
The part that I've found troubling is that within the Oculus Go/Quest Overlay documentation it states, "Equirect layers are single textures that are wrapped into a sphere and projected to surround the user’s view. Most commonly used for 360/180 video playback". I was sure that this meant that it was proven and tested to work for 360 content. If it wasn't obvious by the length and detail of this post, i've wasted many hours trying to get this working because of the documentation stated that it works.
I hope that any future developers happen to find this post before they go down the same path as I did. If this is correct, I also hope that oculus removes the part about 360 videos from their overlay documentation as it simply is not correct.
Just a note that I've spent a lot of time rendering 360- and 180-degree video to Oculus overlay layers, and I haven't experienced this issue. I'm not sure what might be going wrong in your example, but I can at least tell you that it's very possible to have a seamless playback experience.
I've worked at Pixvana since 2016, and we make an online encoding and streaming platform for VR video. I've been primarily responsible for video playback, and our player app, SPIN Play, uses overlay layers for its 360- and 180-degree content. If you have an Oculus Go, download SPIN Play from the Oculus store to see some examples. We just announced that we're discontinuing this platform, but you can still view some of the example videos. I encourage you to find a seam in any of the content.