Forum Discussion

🚨 This forum is archived and read-only. To submit a forum post, please visit our new Developer Forum. 🚨
PJBHL's avatar
PJBHL
Honored Guest
10 months ago

How to detect user interaction and dynamically set a video panel resolution?

Hi,
I'm currently working on a project using the Meta Spatial SDK, and I have a couple of questions that I need help with.

1) Detecting User Interaction to Show a Panel:

In my project, I have a 360° video player panel, built with Jetpack Compose, that disappears after 7 seconds of inactivity. I want it to become visible again whenever there is any user interaction (such as any left or right controller input, pinching with hand tracking). Additionally, I’d like the panel to follow the user’s view (camera orientation) during video playback, so that it always stays in the user's line of sight while watching the 360° video.

Here’s my current implementation:

fun ExoPlayerControlPanel(viewModel: ExoPlayerViewModel) {
    var isPanelVisible = remember { mutableStateOf(true) }
    var lastInteractionTime = remember { mutableLongStateOf(System.currentTimeMillis()) }
    var volume = remember { mutableFloatStateOf(1.0f) }

    // Hide the panel after 7 seconds of inactivity
    LaunchedEffect(isPanelVisible.value) {
        while (isPanelVisible.value) {
            delay(7000)
            val elapsedTime = System.currentTimeMillis() - lastInteractionTime.longValue
            if (elapsedTime >= 7000) {
                Log.d(TAG, "Panel hidden due to inactivity")
                isPanelVisible.value = false
                SpatialActivityManager.executeOnVrActivity<ImmersiveActivity> { activity ->
                    activity.videoPanel!!.setComponent(Visible(false))
                }
            }

            if (some interaction) { // <-- How to properly detect interaction here?
                showPanel
            }
        }
    }

    // Content to show
    Box(
        modifier = Modifier
            .fillMaxSize()
            .background(Color.Black.copy(alpha = 0.8f))
    ) {
        if (isPanelVisible.value) {
          ...
        }
    }
}


What I've tried so far:

  • Adding an event listener to the Sky Video Panel, but it doesn't seem clickable. The event listener works fine on other objects in the scene, but not when the video is playing. This is likely because all the other objects in the scene are disabled during the video playback, leaving only the video visible.
skyVideoPanel?.registerEventListener<ButtonDownEventArgs>(ButtonDownEventArgs.EVENT_NAME) {
    entity: Entity,
    args: ButtonDownEventArgs ->

    if (args.button == ControllerButton.RightSqueeze) {
        val rightController = Query.where { has(AvatarAttachment.id) }
            .eval()
            .filter { it.getComponent<AvatarAttachment>().type == "right_controller" }
            .first()

        videoPanel?.setComponent(Visible(true))
    }
}
  • Instantiating the controllers inside the onCreate method of the scene to detect clicks, but I wasn't able to get it working.

2) Dynamically Setting the Video Panel Resolution:

I also need to dynamically set the video resolution for the panel, depending on the video resolution being played. For example, I want the panel size to adjust based on whether the video is in 4K, 8K, or other resolutions.

Here’s the code I have for registering the panel:

private fun skyVideoPanelRegistration(): PanelRegistration {
    return PanelRegistration(R.integer.sky_video_id) {
        layoutResourceId = R.layout.sky_video_layout
        config {
            // Dynamically change this size depending on the video resolution.
            layoutWidthInPx = 3840
            layoutHeightInPx = 2160

            pivotOffsetWidth = 0.0f
            pivotOffsetHeight = 0.0f

            layerConfig = LayerConfig()
            panelShapeType = PanelShapeType.EQUIRECT
            radiusForCylinderOrSphere = 300.0f

            mips = 1
        }
        panel {
            setIsVisible(false)

            playerView = rootView?.findViewById<PlayerView>(R.id.player_view)
            playerView?.useController = false
            entity?.setComponent(Transform.build { rotateY(180f) })
        }
    }
}

fun playVideo(videoUrl: String) {
    Log.d(TAG, "Video URL: $videoUrl")

    sceneState = SceneState.MOVIE

    playerView?.player = exoPlayer
    val uri = Uri.parse(videoUrl)
    val mediaItem = MediaItem.fromUri(uri)
    exoPlayer!!.addMediaItem(mediaItem)
    exoPlayer!!.prepare()
    exoPlayer!!.playWhenReady = true
}

 

Question:

  • What is the best way to detect any user interaction (left or right controller input, pinching with hand tracking) to make the panel visible again?
  • How can I make the ExoPlayer panel follow the user's view during playback?
  • How can I dynamically adjust the video panel resolution based on the video being played (e.g., 4K, 8K) and change it dynamically during playback if the video quality changes?

Thank you in advance for your help!

Replies have been turned off for this discussion