Issues with Gaussian Splatting integration in Meta Spatial SDK (v0.9.2) – Standalone Quest 3

Have you tested other Gaussian Splatting implementations? There's a few open source projects on Github that have been trying to achieve a similar result

I'm sharing the specific code from my ImmersiveActivity.kt for a more detailed analysis. Here is the relevant code structure:

package com.example.apptest

import android.annotation.SuppressLint
import android.os.Bundle
import android.view.View
import android.webkit.WebView
import android.widget.TextView
import androidx.compose.ui.platform.ComposeView
import androidx.core.net.toUri
import com.meta.spatial.castinputforward.CastInputForwardFeature
import com.meta.spatial.compose.ComposeFeature
import com.meta.spatial.compose.ComposeViewPanelRegistration
import com.meta.spatial.core.BuildConfig
import com.meta.spatial.core.Entity
import com.meta.spatial.core.Pose
import com.meta.spatial.core.SpatialFeature
import com.meta.spatial.core.SpatialSDKExperimentalAPI
import com.meta.spatial.core.Vector3
import com.meta.spatial.datamodelinspector.DataModelInspectorFeature
import com.meta.spatial.debugtools.HotReloadFeature
import com.meta.spatial.isdk.IsdkFeature
import com.meta.spatial.okhttp3.OkHttpAssetFetcher
import com.meta.spatial.ovrmetrics.OVRMetricsDataModel
import com.meta.spatial.ovrmetrics.OVRMetricsFeature
import com.meta.spatial.runtime.NetworkedAssetLoader
import com.meta.spatial.runtime.SceneMaterial
import com.meta.spatial.toolkit.AppSystemActivity
import com.meta.spatial.toolkit.DpPerMeterDisplayOptions
import com.meta.spatial.toolkit.LayoutXMLPanelRegistration
import com.meta.spatial.toolkit.Material
import com.meta.spatial.toolkit.Mesh
import com.meta.spatial.toolkit.MeshCollision
import com.meta.spatial.toolkit.PanelRegistration
import com.meta.spatial.toolkit.PanelStyleOptions
import com.meta.spatial.toolkit.QuadShapeOptions
import com.meta.spatial.toolkit.Transform
import com.meta.spatial.toolkit.UIPanelSettings
import com.meta.spatial.vr.VRFeature
import java.io.File
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.launch
import com.meta.spatial.splat.Splat
import com.meta.spatial.splat.SplatFeature
import android.net.Uri
import com.meta.spatial.splat.SpatialSDKExperimentalSplatAPI


class ImmersiveActivity : AppSystemActivity() {
  private val activityScope = CoroutineScope(Dispatchers.Main)

  lateinit var textView: TextView
  lateinit var webView: WebView

  @OptIn(SpatialSDKExperimentalSplatAPI::class)
  override fun registerFeatures(): List<SpatialFeature> {
    val features =
        mutableListOf<SpatialFeature>(
            VRFeature(this),
            SplatFeature(),
            ComposeFeature(),
            IsdkFeature(this, spatial, systemManager),
        )
    if (BuildConfig.DEBUG) {
      features.add(CastInputForwardFeature(this))
      features.add(HotReloadFeature(this))
      features.add(OVRMetricsFeature(this, OVRMetricsDataModel() { numberOfMeshes() }))
      features.add(DataModelInspectorFeature(spatial, this.componentManager))
    }
    return features
  }

  override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    NetworkedAssetLoader.init(
        File(applicationContext.getCacheDir().canonicalPath),
        OkHttpAssetFetcher(),
    )

    loadGLXF()
  }

  override fun onSceneReady() {
    super.onSceneReady()

    scene.setLightingEnvironment(
        ambientColor = Vector3(0f),
        sunColor = Vector3(7.0f, 7.0f, 7.0f),
        sunDirection = -Vector3(1.0f, 3.0f, -2.0f),
        environmentIntensity = 0.3f,
    )
    scene.updateIBLEnvironment("environment.env")

    scene.setViewOrigin(0.0f, 0.0f, 2.0f, 180.0f)

    // Creating Skybox
    Entity.create(
        listOf(
            Mesh("mesh://skybox".toUri(), hittable = MeshCollision.NoCollision),
            Material().apply {
              baseTextureAndroidResourceId = R.drawable.skydome
              unlit = true // Prevent scene lighting from affecting the skybox
            },
            Transform(Pose(Vector3(x = 0f, y = 0f, z = 0f))),
        )
    )
      Entity.create(
          listOf(
              // Assurez-vous qu'un fichier comme "room.splat" existe bien dans app/src/main/assets/
              Splat("apk:///assets/room.splat".toUri()),

              // Positionne et met à l'échelle le splat dans la scène
              Transform(
                  Pose(
                      position = Vector3(x = 0f, y = 1.5f, z = -2f), // Position: 1.5m de haut, 2m devant la caméra
                      scale = Vector3(x = 1f, y = 1f, z = 1f)
                  )
              )
          )
      )
  }


  fun playVideo(webviewURI: String) {
    textView.visibility = View.GONE
    webView.visibility = View.VISIBLE
    val additionalHttpHeaders = mapOf("Referer" to "https://${packageName}")
    webView.loadUrl(webviewURI, additionalHttpHeaders)
  }

  @OptIn(SpatialSDKExperimentalAPI::class)
  override fun registerPanels(): List<PanelRegistration> {
    return listOf(
        // Registering light-weight Views panel
        LayoutXMLPanelRegistration(
            R.id.ui_example,
            layoutIdCreator = { _ -> R.layout.ui_example },
            settingsCreator = { _ -> UIPanelSettings() },
            panelSetupWithRootView = { rootView, _, _ ->
              webView =
                  rootView.findViewById<WebView>(R.id.web_view) ?: return@LayoutXMLPanelRegistration
              textView =
                  rootView.findViewById<TextView>(R.id.text_view)
                      ?: return@LayoutXMLPanelRegistration
              val webSettings = webView.settings
              @SuppressLint("SetJavaScriptEnabled")
              webSettings.javaScriptEnabled = true
              webSettings.mediaPlaybackRequiresUserGesture = false
            },
        ),
        // Registering a Compose panel
        ComposeViewPanelRegistration(
            R.id.options_panel,
            composeViewCreator = { _, context ->
              ComposeView(context).apply { setContent { OptionsPanel(::playVideo) } }
            },
            settingsCreator = {
              UIPanelSettings(
                  shape =
                      QuadShapeOptions(width = OPTIONS_PANEL_WIDTH, height = OPTIONS_PANEL_HEIGHT),
                  style = PanelStyleOptions(themeResourceId = R.style.PanelAppThemeTransparent),
                  display = DpPerMeterDisplayOptions(),
              )
            },
        ),
    )
  }

  override fun onSpatialShutdown() {
    super.onSpatialShutdown()
  }

  private fun loadGLXF(): Job {
    return activityScope.launch {
      glXFManager.inflateGLXF(
          "apk:///scenes/Composition.glxf".toUri(),
          keyName = "example_key_name",
          onLoaded = { glxfInfo ->
            // get the environment mesh and set it to use an unlit shader.
            val environmentEntity: Entity = glxfInfo.getNodeByName("Environment").entity
            val environmentMesh = environmentEntity.getComponent<Mesh>()
            environmentMesh.defaultShaderOverride = SceneMaterial.UNLIT_SHADER
            environmentEntity.setComponent(environmentMesh)
          },
      )
    }
  }
}

Here the red problems :
No value passed for parameter 'context'.
No value passed for parameter 'systemManager'.
No parameter with name 'position' found.
No parameter with name 'scale' found.

Thank you for your guidance.

Hi valtik! 

Could you share any more detail on the reference errors or the code snippets you are using to add the splats to the scene? 

You can also take a look at the Splat Sample for an example of how to use it: https://github.com/meta-quest/Meta-Spatial-SDK-Samples/tree/main/SplatSample

Hi everyone,

I've been working on improving the rendering quality of the Gaussian Splatting sample in the Meta Spatial SDK, and I’m looking for ways to bridge the gap between the current sample fidelity and the results seen in Meta Horizon Hyperscape.

I have a few technical questions regarding the roadmap and current capabilities:

1. Cloud Rendering for Custom Apps: Hyperscape’s quality is impressive, likely due to cloud-based processing or streaming. Is there a plan to make a similar cloud-rendering backend available for developers using the Spatial SDK for their own standalone Android apps?

2. Spherical Harmonics (SH) Control: Currently, it appears that SH is locked at Degree 0 in the SDK, resulting in a "flat" look without the view-dependent reflections seen on PC viewers. Is there a way to unlock or adjust the SH Degree (e.g., to Degree 1 or 2) to better suit specific high-fidelity needs?

3. Splat Stretching & Scaling: I’ve noticed that splats often appear more "stretched" and blurry in the headset compared to the exact same .spz file viewed on a PC. This suggests a difference in how the rasterizer handles splat scaling or alpha blending on the mobile chipset.

Could we get access to more granular quality settings (such as Splat Scale factor, SH Degree, or Sorting depth) to find the "sweet spot" for specific datasets?

Are these features currently on the roadmap for a future update of the Meta Spatial SDK?Thanks for your help!

I am running into the exact same usage case, and I was using Aras' implementation until I found out its performance on VR.

How is your experience with the Spatial SDK's imported Gaussians so far? Are they visually compelling? I think Meta did not open-source their implementation, so I cannot tell what optimizations and new techniques they use, which is a pity since I want to reproduce the SOTA work based on Aras, but Meta seems to already done that.