360 Video Streaming Performance on Quest 2 and Quest 3

Hi,

Our team is currently in the process of studying and porting a project previously developed in Unity to the new Meta Spatial SDK. This project involves streaming monoscopic 360° videos (not 3D stereoscopic) on the Quest 2 and Quest 3. I'm using the MediaPlayerSample project available on GitHub as a reference for studying. For video playback, I'm utilizing ExoPlayer for streaming, as VideoView, which was initially used in the project, showed significant performance issues during streaming.

While testing, I noticed the following performance differences:

• On the Quest 2, videos with resolutions starting at 4K and above cause severe performance degradation. The device often freezes, and the application becomes unstable, with frame rates dropping to between 30–50 FPS.
• On the Quest 3, videos up to 5K resolution run perfectly at 90 FPS without any performance issues.

This raises a few questions:

1. Is there a way to detect programmatically (in Kotlin) whether the user is using a Quest 2 or Quest 3? Currently, to address performance issues on the Quest 2, I’m dividing the video resolution by 1.75. If I can’t find a solution to improve performance on the Quest 2, I may have to keep this adjustment exclusively for Quest 2 devices.
2. Are there specific considerations for implementing shaders (.frag and .vert) for rendering high-resolution 360° videos?
3. What is the recommended video format and encoding for streaming monoscopic 360° videos, particularly to optimize performance on the Quest 2?

Below are the shaders I’m currently using. Is there’s anything wrong or missing in my implementation?

Fragment Shader (.frag):

# 360.frag
#version 400
#extension GL_ARB_separate_shader_objects : enable
#extension GL_ARB_shading_language_420pack : enable

#include <common.glsl>

layout (std140, set = 3, binding = 0) uniform MaterialUniform {
  vec4 stereoParams;
  vec4 customParams;
} g_MaterialUniform;

layout (set = 3, binding = 1) uniform sampler2D albedoSampler;

layout(location = 0) in struct {
  vec4 color;
  vec2 albedoCoord;
  vec2 roughnessMetallicCoord;
  vec2 emissiveCoord;
  vec2 occlusionCoord;
  vec2 normalCoord;
  vec3 lighting;
  vec3 worldNormal;
  vec3 worldPosition;
} vertexOut;

layout (location = 0) out vec4 outColor;

void main() {
  vec4 pixel = texture(albedoSampler, vertexOut.albedoCoord);

  outColor.rgba = pixel;
}

 Vertex Shader (.vert):

# 360.vert
#version 430
#extension GL_ARB_separate_shader_objects : enable
#extension GL_ARB_shading_language_420pack : enable

#include <common.glsl>
#include <app2vertex.glsl>

layout(location = 0) out struct {
  vec4 color;
  vec2 albedoCoord;
  vec2 roughnessMetallicCoord;
  vec2 emissiveCoord;
  vec2 occlusionCoord;
  vec2 normalCoord;
  vec3 lighting;
  vec3 worldNormal;
  vec3 worldPosition;
} vertexOut;

layout (std140, set = 3, binding = 0) uniform MaterialUniform {
  vec4 stereoParams;
  vec4 customParams;
} g_MaterialUniform;

void main() {
  App2VertexUnpacked app = getApp2VertexUnpacked();

  vec4 wPos4 = g_PrimitiveUniform.worldFromObject * vec4(app.position, 1.0f);

  vertexOut.albedoCoord = app.uv;
  vertexOut.lighting = app.incomingLighting;
  vertexOut.worldPosition = wPos4.xyz;
  vertexOut.worldNormal = normalize((transpose(g_PrimitiveUniform.objectFromWorld) * vec4(app.normal, 0.0f) ).xyz);

  gl_Position = getClipFromWorld() * wPos4;

  postprocessPosition(gl_Position);
}

Additionally, the video is being rendered using the following configuration:

return PanelRegistration(R.integer.sky_video_id) {
    layoutResourceId = R.layout.sky_video_layout
    config {
        val SPHERE_RADIUS: Float = 300.0f

        // Some videos will have other resolutions, this is just a base to test performance.
        layoutWidthInPx = 5760
        layoutHeightInPx = 2880

        pivotOffsetWidth = 0.0f
        pivotOffsetHeight = 0.0f

        // Do i need stereoMode here ?
        stereoMode = StereoMode.MonoUp

        sceneMeshCreator = { texture: SceneTexture ->
            skyPanelMaterial!!.apply { setTexture("albedoSampler", texture) }
            SceneMesh.skybox(SPHERE_RADIUS, skyPanelMaterial!!)
        }
    }
    panel {
        setIsVisible(false)
        playerView = rootView?.findViewById<PlayerView>(R.id.player_view)
        entity?.setComponent(Transform.build { rotateY(180f) })
    }
}

Thank you in advance for your help!