Setting up Multimodal hands & controllers

[nbrindic.oculus.dev]

Hi,

I am trying to run Multimodal feature outlined in Meta recent blog posts (https://www.uploadvr.com/meta-quest-pro-simultaneous-hands-controllers-mode/).

My device is Quest Pro with OS version 57.0.0.261.669 and Quest Pro controllers. I am working in Unity 2022.3.10.

So far, I am failing to make this feature work. The settings of my OVRManager are:

When I try to run it on device, I am getting the following message in the logs:

"Failed to set multimodal hands and controllers mode!"

Do you have any idea what I am doing wrong?

[Almxdena]

Hey!

I am experiencing a similar issue. I think I set up the multimodal hands and controllers option the same way, but the result that I get is seeing my hands or the controllers, not both at the same time. I got to see my left hand if I hold the right controller with the right hand (that immediately changes to the controller), but I am not able to see my hand while holding the controller ad the video of the blog...

[Fahruz]

I'm also wondering how to enable this new feature. I imagine the reason it fails in Unity is because it first has to be activated in the headset or via the Oculus app, but I couldn't find any "Multimodal" feature in the settings, including in "Experimental".

[Almxdena]

Yes, I am having the same problem. Moreover, I have the same software version and meta headset. I will still try to figure out how this work. If I do, I will tell you! Please, do the same.

[gedo831]

Does anyone find the solution for this issue? Now I am trying to make it work with SDK v57 but still has no luck. I cannot run the "MultimordalHandsAndControllers" sample scene properly.

[markeahogan]

Do you have experimental enabled on your device?

adb shell setprop debug.oculus.experimentalEnabled 1

 and experimental ticked on in the OVRManager might be important too 

[gedo831]

Hey, thank you for your advice. Now it seems working somehow in my project. I did not notice that was one of the experimental feature.

[Almxdena]

I did, but it doesn't work as the gif that you can see here: https://www.uploadvr.com/meta-quest-pro-simultaneous-hands-controllers-mode/ (which is seeing the controller while using the hand tracking). For me, it works as what you see in this video... https://drive.google.com/file/d/173sddlVS7bVM4CDwteS89hEetA_dS3_2/view?usp=share_link

First, I start with both controllers, but few seconds after starting the video, I quit the controller from the right hand. Hence, as you can observe, I am using the left controller, and the right hand tracking. Everything is correctly working, but not as expected. I want to see the controllers at the same time, as shown in the gif of the other website. Why? Because for me, it would be very interesting to use the controller as an object tracker.

[Almxdena]

I wrote a script to see the 3D models of the controllers all the time, but the position doesn't update if you're not using them...

[gedo831]

Almxdena, you're right. Initially, I was considering using the controller as an object tracker. While I did succeed in a certain kind of simultaneous use of hand tracking and controller tracking, I couldn't achieve the true simultaneous behavior as shown in your linked video. Currently, what I can achieve is a concurrent simultaneous use where one hand uses hand tracking and the other uses controller tracking, with proper mesh rendering. I can't track the released controller. If it's possible to track the released controller as a 6DOF tracker in the state shown in the linked video, I'd love to know how, and I believe that's the functionality developers desire. However, even in the linked video, the released controller isn't moving. This means that they might just be displaying the controller mesh on the table the moment the controller is released and hand tracking is activated.