Forum Discussion

vivekrajagopal's avatar
4 years ago
Solved

Meta Avatars 2 Lipsync - PUN 2 & Photon Voice issue!

Dear Devs,

 

I'm struggling with problem since a week. I use PUN 2 and Photon Voice to bring Meta Avatars 2 in a multiplayer environment. 

Below are my issues,

1. When there is no Photon Voice setup in the scene, the Meta Avatars lipsync works perfect in the Photon multiplayer room.

2. When I add the Photon Voice to the prefab and setup the scene with Photon Voice Network, only the voice is there the Meta Avatars lipsync does not work. 

 

I understand there is a race condition happening between these two plugins. 

 

Please kindly help me resolve if anyone has already resolved such problem. This thread can help other devs as well in the future.

 

Thanks!

29 Replies

  • Sorry, I couldn't find a workaround either, so I ended up disabling lip-sync. Ideally there would be a checkbox on the avatar system to allow PUN to also access the microphone.

  • olaysia's avatar
    olaysia
    Expert Protege

    I found a solution to this:

     

    The Problem:

    The scripts UnityMicrophone(by Photon) and LipSyncMicInput(part of the avatars 2 sdk) both contain this line 

     

     

    Microphone.Start(deviceName, looping, audioClipLength, samplingRate). 

     

     

    This method creates a new audio clip which gets filled with the input from your microphone. This causes the following situation:

    1. UnityMicrophone creates a new audio clip for itself using the Microphone.Start() function and is happily using an audio clip which is being continuously filled with the input of your microphone.
    2. Now LipSyncMicInput also calls the Microphone.Start(), now the input of your microphone is being stolen by LipSyncMicInput's audio clip and the audio clip which Photon's UnityMicrophone was using now helplessly loops over itself.
    3. The same can happen the other way around, it just depends who called Microphone.Start() first.

    In summary, the audio clip which LipSyncMicInput or Photon.Recorder  is using will be useless as soon as one of them calls Microphone.Start().

     

    My Solution:

    1. I created a central Microphone object which would have the sole responsibility of Starting and stopping the Microphone.
    2. To deal with Photon, I changed the Input source from Microphone to Factory. And created a custom input source which used my central microphone as its input. 
    3. To deal with LipSync, I REMOVED LipSyncMicInput from my LipSyncInput object, and replaced it with a script which would continuously check the LipSyncInput object's audio source clip against my central Microphone object's audio clip. If different, it would update the lip sync's audio clip to be the same as my central Microphone's. I also had to force the clip's position to match that of my microphone's.

    For help creating a custom input source for the Photon Recorder I recommend just googling it.

    Here's the script I used to replace LipSyncMicInput, GGMicrophone is my central microphone object.

     

     

    public class RuntimeAudioClipUpdater : MonoBehaviour
    {
        AudioSource audioSource;
    
        private void Start()
        {
            audioSource = GetComponent<AudioSource>();
        }
    
        void Update()
        {
            if (audioSource != null)
            {
                AudioClip clip = audioSource.clip;
                AudioClip referenceClip = GGMicrophone.Instance.GetMicrophoneAudioClip();
                int referenceClipId = referenceClip.GetInstanceID();
                if (clip != null)
                {
                    int currentClipId = clip.GetInstanceID();
                    if (currentClipId != referenceClipId)
                    {
                        //audioSource.Stop();
                        audioSource.clip = referenceClip;
                        audioSource.Play();
                    }
                }
                else
                {
                    //audioSource.Stop();
                    audioSource.clip = referenceClip;
                    audioSource.Play();
                }
            }
    
            audioSource.timeSamples = GGMicrophone.Instance.GetPosition();
        }
    
    
    }

     

     

     

  • Hi, i'm trying to do as you did with your code but it keeps giving me the
    NullReferenceExeprion: Object reference not set to an istance of an object 
    RuntimeAudioClipUpdater.Update() (RuntimeAudioClipUpdater.cs.20).

     

     



    What i did is:
    1) Changing the Recorder Microphone Type to Factory

     


    2) (LipSyncInput) Removing Lip Sync Mic Input and setting Ovr Avataer Lip Sync Context

     


    3) (LipSyncInput) Adding Runtime Audio Clip Updater
    4) Cretaing a new empty object than call GGMicrophoneInputForPhotonInitializer

     



    Am i missing something, or doing something wrong? The script are the same as the one you pubblished

    • olaysia's avatar
      olaysia
      Expert Protege

      You may have to tweak things a bit for your particular solution. But let me see if I can help. If you click on the null reference exception, which line of the script RuntimeAudioClipUpdater is it complaining about?

      • Everysimo's avatar
        Everysimo
        Explorer

        Hi, sorry i forgot to share the line of code

        AudioClip referenceClip = GGMicrophone.Instance.GetMicrophoneAudioClip();