Skip to main content
Inspiring
May 26, 2022
Solved

Meta Avatars 2 Lipsync - PUN 2 & Photon Voice issue!

Dear Devs,

 

I'm struggling with problem since a week. I use PUN 2 and Photon Voice to bring Meta Avatars 2 in a multiplayer environment. 

Below are my issues,

1. When there is no Photon Voice setup in the scene, the Meta Avatars lipsync works perfect in the Photon multiplayer room.

2. When I add the Photon Voice to the prefab and setup the scene with Photon Voice Network, only the voice is there the Meta Avatars lipsync does not work. 

 

I understand there is a race condition happening between these two plugins. 

 

Please kindly help me resolve if anyone has already resolved such problem. This thread can help other devs as well in the future.

 

Thanks!

This topic has been closed for replies.
Inspiring
January 8, 2023

Dear Developers, 

I've found a proper fix for this issue. I've explained it in my blog that might help you. 

Please check this blog: https://medium.com/@vivekrajagopal_84414/meta-avatars-2-and-photon-unity-voice-issue-lipsync-pun-voice-resolved-6a7c5d229b1c

29 replies

ezone
Up and Coming
September 20, 2022

Sorry, I couldn't find a workaround either, so I ended up disabling lip-sync. Ideally there would be a checkbox on the avatar system to allow PUN to also access the microphone.

Simon Edis - Ezone.com
Inspiring
October 3, 2022

I found a solution to this:

 

The Problem:

The scripts UnityMicrophone(by Photon) and LipSyncMicInput(part of the avatars 2 sdk) both contain this line 

 

 

Microphone.Start(deviceName, looping, audioClipLength, samplingRate). 

 

 

This method creates a new audio clip which gets filled with the input from your microphone. This causes the following situation:

  1. UnityMicrophone creates a new audio clip for itself using the Microphone.Start() function and is happily using an audio clip which is being continuously filled with the input of your microphone.
  2. Now LipSyncMicInput also calls the Microphone.Start(), now the input of your microphone is being stolen by LipSyncMicInput's audio clip and the audio clip which Photon's UnityMicrophone was using now helplessly loops over itself.
  3. The same can happen the other way around, it just depends who called Microphone.Start() first.

In summary, the audio clip which LipSyncMicInput or Photon.Recorder  is using will be useless as soon as one of them calls Microphone.Start().

 

My Solution:

  1. I created a central Microphone object which would have the sole responsibility of Starting and stopping the Microphone.
  2. To deal with Photon, I changed the Input source from Microphone to Factory. And created a custom input source which used my central microphone as its input. 
  3. To deal with LipSync, I REMOVED LipSyncMicInput from my LipSyncInput object, and replaced it with a script which would continuously check the LipSyncInput object's audio source clip against my central Microphone object's audio clip. If different, it would update the lip sync's audio clip to be the same as my central Microphone's. I also had to force the clip's position to match that of my microphone's.

For help creating a custom input source for the Photon Recorder I recommend just googling it.

Here's the script I used to replace LipSyncMicInput, GGMicrophone is my central microphone object.

 

 

public class RuntimeAudioClipUpdater : MonoBehaviour
{
    AudioSource audioSource;

    private void Start()
    {
        audioSource = GetComponent<AudioSource>();
    }

    void Update()
    {
        if (audioSource != null)
        {
            AudioClip clip = audioSource.clip;
            AudioClip referenceClip = GGMicrophone.Instance.GetMicrophoneAudioClip();
            int referenceClipId = referenceClip.GetInstanceID();
            if (clip != null)
            {
                int currentClipId = clip.GetInstanceID();
                if (currentClipId != referenceClipId)
                {
                    //audioSource.Stop();
                    audioSource.clip = referenceClip;
                    audioSource.Play();
                }
            }
            else
            {
                //audioSource.Stop();
                audioSource.clip = referenceClip;
                audioSource.Play();
            }
        }

        audioSource.timeSamples = GGMicrophone.Instance.GetPosition();
    }


}

 

 

 

Participant
November 2, 2022
Hi Olaysia, that is a great solution! Thanks for sharing.
 
I have been struggling to use a custom audio source with PUN voice 2 Recorder, based on the documentation. 
 
Is there any chance to share your implementation. 
 
Thank you! 
Inspiring
November 2, 2022

Sure, which part would you like me to share? The Central Microphone, the custom photon recorder input or the lips sync clip updater?

Participant
January 5, 2023

Hi, i'm trying to do as you did with your code but it keeps giving me the
NullReferenceExeprion: Object reference not set to an istance of an object 
RuntimeAudioClipUpdater.Update() (RuntimeAudioClipUpdater.cs.20).

 

 



What i did is:
1) Changing the Recorder Microphone Type to Factory

 


2) (LipSyncInput) Removing Lip Sync Mic Input and setting Ovr Avataer Lip Sync Context

 


3) (LipSyncInput) Adding Runtime Audio Clip Updater
4) Cretaing a new empty object than call GGMicrophoneInputForPhotonInitializer

 



Am i missing something, or doing something wrong? The script are the same as the one you pubblished

Inspiring
January 5, 2023

You may have to tweak things a bit for your particular solution. But let me see if I can help. If you click on the null reference exception, which line of the script RuntimeAudioClipUpdater is it complaining about?

Participant
January 5, 2023

Hi, sorry i forgot to share the line of code

AudioClip referenceClip = GGMicrophone.Instance.GetMicrophoneAudioClip();

 

vivekrajagopalAuthorAnswer
Inspiring
January 8, 2023

Dear Developers, 

I've found a proper fix for this issue. I've explained it in my blog that might help you. 

Please check this blog: https://medium.com/@vivekrajagopal_84414/meta-avatars-2-and-photon-unity-voice-issue-lipsync-pun-voice-resolved-6a7c5d229b1c

Inspiring
January 9, 2023

Nice! That's a better solution to what I've got.

New Participant
November 8, 2024

Nice 

New Participant
November 8, 2024

Good