Question about the lip-sync for the new expressive avatar

[ericwang0701]

Hi,
I have followed the developer guide for the new expressive avatar update and every thing works well except the lip-sync feature.
In this document, it only mentioned that the expressive avatar utilized the OVRLipsync to enable realistic lip-syncing. However, it didn't mention how we actually make it work for the expressive avatar (e.g. what prefab or scripts that we should looked at and attach on the avatar)

Any help or suggestion would be really appreciated.

Best Regards,
Eric Wang

[legolasshegolas]

@stevehinan
Yes, please. The head doesn't turn either. The hands are moving with the controller.

[legolasshegolas]

Okay, I have avatar working. Head and hands have movement.
For the head to move, I had to set Texture Compression to ASTC in Build Settings. But one problem that has risen from this is that the avatar disappears on it's own in a few seconds after the app is launched.

I still don't have the LipSync working. Don't know how to get the LipSync library working with the Avatar lib. Resources about how to get that working would be appreciated!

[Hexapuss_Steve]

@legolasshegolas

I have encountered a lipsync issue that might be similar:

When I freshly install my app on Oculus Go and grant microphone permissions via popup, I don't see the lipsync working.  But when I close the app and start it up again the lipsync works as expected.  

[cloud_canvas]

My app has a lot of issues associated with that as well. Unity only recently added support for Android runtime permission requests, and their implementation has a few bugs. In the case of RECORD_AUDIO (used for microphone access), you can't stuff the request up top before the app launches anymore where you used to be able to.

You may want to pin your use of microphone functionality around a coroutine or an async Task that uses WaitUntil() that is waiting for the user to grant microphone permission to do whatever initialization behavior is necessary. (I guess in LipSync?)
I hadn't been having the same issue as you guys, but it's probably because my app is using Vivox which is asking for the microphone permission well before an OvrAvatar would have spawned and thus fired up the LipSync .aars to tap into the mic.

[legolasshegolas]

So I'm trying to follow the instructions here to try to get the lipsync working: https://developer.oculus.com/documentation/audiosdk/latest/concepts/audio-ovrlipsync-sample-details/

I can't figure this part out:
"OVRLipSyncContext must be added to each GameObject which has the morph or texture target that you want to control.

OVRLipSyncContextMorphTarget and OVRLipSyncContextTextureFlip are the scripts that bridge the viseme output from OVRLipSyncContext."

Do avatars even have facial blendshapes? If yes, how do I access them for the OVRLipSyncContextMorphTarget script?

[legolasshegolas]

Also, why does the LocalAvatar gameobject disappear for me in a couple of seconds when I have Texture Compression set to ASTC in Build Settings

[Hexapuss_Steve]

@legolasshegolas
If you are using a recent Unity Oculus Ingegration you should be able to do all the lipsync within the OVRAvatar script via the inspector.  Check enable Expressive, Can own microphone, Enable mouth vertex animation.
I believe that's all you need to do, as long as you have the Oculus Package imported into your project including the LipSync folder.  Here is a screengrab of my OVRAvatar settings in Unity, note the AudioSource. 

[legolasshegolas]

@stevehinan Thank you for the reply!
Okay so what was causing the trouble was the OculusGo's microphone not working... >.> Restarted the headset and lipsync is working. 

Still, the avatar disappears after a few seconds. What causes this is Texture Compression set to ASTC in Build Settings. Setting it to anything else makes the avatar's head to lose all motion. Doesn't disappear though.

Also, how can I get the avatar lipsync to work with an audio file rather than live input from the microphone?

[Hexapuss_Steve]

@legolasshegolas
To troubleshoot the avatar disappearing I would create a new project and use the Oculus provided prefabs to see if that works as expected, if yes - see what is different with your project, if no - maybe a project setting?

LipSync from an audio file, I have not tried this before.  Please share if you find a solution.

[GigaSora]

For anyone still having trouble after all that, here's what I've found after a little ablation testing. For Lip-sync to work out of box with the avatar system, you must:
- have enable expressive selected
- can own microphone selected
- game view must be selected if in editor, no scene view or even clicking away
- your local avatar object must have the ovr local avatar driver component on it
- It does not appear you can have multiple local avatar drivers in the scene, though there was some ambiguity around that