Meta Avatars 2 Lipsync - PUN 2 & Photon Voice issue!

[vivekrajagopal]

Dear Devs,

I'm struggling with problem since a week. I use PUN 2 and Photon Voice to bring Meta Avatars 2 in a multiplayer environment. 

Below are my issues,

1. When there is no Photon Voice setup in the scene, the Meta Avatars lipsync works perfect in the Photon multiplayer room.

2. When I add the Photon Voice to the prefab and setup the scene with Photon Voice Network, only the voice is there the Meta Avatars lipsync does not work. 

I understand there is a race condition happening between these two plugins. 

Please kindly help me resolve if anyone has already resolved such problem. This thread can help other devs as well in the future.

Thanks!

[ezone]

Sorry, I couldn't find a workaround either, so I ended up disabling lip-sync. Ideally there would be a checkbox on the avatar system to allow PUN to also access the microphone.

[olaysia]

I found a solution to this:

The Problem:

The scripts UnityMicrophone(by Photon) and LipSyncMicInput(part of the avatars 2 sdk) both contain this line 

Microphone.Start(deviceName, looping, audioClipLength, samplingRate). 

This method creates a new audio clip which gets filled with the input from your microphone. This causes the following situation:

1. UnityMicrophone creates a new audio clip for itself using the Microphone.Start() function and is happily using an audio clip which is being continuously filled with the input of your microphone.
2. Now LipSyncMicInput also calls the Microphone.Start(), now the input of your microphone is being stolen by LipSyncMicInput's audio clip and the audio clip which Photon's UnityMicrophone was using now helplessly loops over itself.
3. The same can happen the other way around, it just depends who called Microphone.Start() first.

In summary, the audio clip which LipSyncMicInput or Photon.Recorder  is using will be useless as soon as one of them calls Microphone.Start().

My Solution:

1. I created a central Microphone object which would have the sole responsibility of Starting and stopping the Microphone.
2. To deal with Photon, I changed the Input source from Microphone to Factory. And created a custom input source which used my central microphone as its input. 
3. To deal with LipSync, I REMOVED LipSyncMicInput from my LipSyncInput object, and replaced it with a script which would continuously check the LipSyncInput object's audio source clip against my central Microphone object's audio clip. If different, it would update the lip sync's audio clip to be the same as my central Microphone's. I also had to force the clip's position to match that of my microphone's.

For help creating a custom input source for the Photon Recorder I recommend just googling it.

Here's the script I used to replace LipSyncMicInput, GGMicrophone is my central microphone object.

public class RuntimeAudioClipUpdater : MonoBehaviour
{
    AudioSource audioSource;

    private void Start()
    {
        audioSource = GetComponent<AudioSource>();
    }

    void Update()
    {
        if (audioSource != null)
        {
            AudioClip clip = audioSource.clip;
            AudioClip referenceClip = GGMicrophone.Instance.GetMicrophoneAudioClip();
            int referenceClipId = referenceClip.GetInstanceID();
            if (clip != null)
            {
                int currentClipId = clip.GetInstanceID();
                if (currentClipId != referenceClipId)
                {
                    //audioSource.Stop();
                    audioSource.clip = referenceClip;
                    audioSource.Play();
                }
            }
            else
            {
                //audioSource.Stop();
                audioSource.clip = referenceClip;
                audioSource.Play();
            }
        }

        audioSource.timeSamples = GGMicrophone.Instance.GetPosition();
    }


}

[ErickMarinC]

Hi Olaysia, that is a great solution! Thanks for sharing.

I have been struggling to use a custom audio source with PUN voice 2 Recorder, based on the documentation. 

https://doc.photonengine.com/en-us/voice/v2/troubleshooting/faq#how_to_use_a_custom_audio_source_

Is there any chance to share your implementation. 

Thank you! 

[olaysia]

Sure, which part would you like me to share? The Central Microphone, the custom photon recorder input or the lips sync clip updater?

[ErickMarinC]

It would be amazing to have:

• Custom Photon Recorder Input
• Central Microphone

Thank you so much!

[olaysia]

This is the script for my custom Photon Recorder Input. I basically just copied Photon's MicrophoneWrapper except, as you can see, instead of calling Microphone.Start() I instead fetch the audio clip from my central microphone with GGMicrophone.GetMicrophoneAudioClip(). 'GG' is just a marker for classes that my company writes.

using UnityEngine;
using Photon.Voice;

public class GGMicrophoneInputForPhoton : IAudioReader<float>
{
    public GGMicrophoneInputForPhoton()
    {
        GGMicrophone.Instance.StartMicrophone();
    }

    public int SamplingRate
    {
        get
        {
            AudioClip clip = GGMicrophone.Instance.GetMicrophoneAudioClip();
            return clip.frequency;
        }
    }

    public int Channels
    {
        get
        {
            AudioClip clip = GGMicrophone.Instance.GetMicrophoneAudioClip();
            return clip.channels;
        }
    }

    public string Error { get; private set; }

    public void Dispose()
    {
        GGMicrophone.Instance.StopMicrophone();
    }

    private int micPrevPos;
    private int micLoopCnt;
    private int readAbsPos;

    public bool Read(float[] buffer)
    {
        if (Error != null)
        {
            return false;
        }
        int micPos = GGMicrophone.Instance.GetPosition();
        // loop detection
        if (micPos < micPrevPos)
        {
            micLoopCnt++;
        }
        micPrevPos = micPos;

        AudioClip clip = GGMicrophone.Instance.GetMicrophoneAudioClip();

        var micAbsPos = micLoopCnt * clip.samples + micPos;

        if (clip.channels == 0)
        {
            Error = "Number of channels is 0 in Read()";
            //logger.LogError("[PV] MicWrapper: " + Error);
            return false;
        }
        var bufferSamplesCount = buffer.Length / clip.channels;

        var nextReadPos = this.readAbsPos + bufferSamplesCount;
        if (nextReadPos < micAbsPos)
        {
            clip.GetData(buffer, this.readAbsPos % clip.samples);
            this.readAbsPos = nextReadPos;
            return true;
        }
        else
        {
            return false;
        }
    }
}

To initialise the custom input I use this very simple script:

using GGChassis.GGAvatars;
using Oculus.Platform;
using Photon.Pun;
using Photon.Voice.PUN;
using Photon.Voice.Unity;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class GGMicrophoneInputForPhotonInitializer : MonoBehaviour
{
    private void Start()
    {
        Recorder recorder = PhotonVoiceNetwork.Instance.PrimaryRecorder;
        if (recorder != null)
        {
            recorder.SourceType = Recorder.InputSourceType.Factory;
            recorder.InputFactory = () => new GGMicrophoneInputForPhoton();
        }
        else
        {
            Debug.LogError("Could not set recorder's input source type because no recorder was found.");
        }
    }
}

And here is my central microphone script:

public class GGMicrophone : MonoBehaviour
{
    [SerializeField] AudioClip m_audioClip;
    [Tooltip("Enable to see the Mic Input Volume")][SerializeField] bool m_debugMicrophone = true;
    [Range(0f, 1f)] [SerializeField] float m_micInputVolume = 0f;
    [Range(0, 1000000)] [SerializeField] int m_microphonePosition;
    [SerializeField] int m_audioClipId;
    [SerializeField] float m_microphoneSensitivity = 50f;
    [SerializeField] float m_threshold = 0.1f;
    int m_samplingRate = 48000;
    int m_sampleWindow = 64;
    int m_audioClipLength = 1;

    static private GGMicrophone m_instance;
    static public GGMicrophone Instance
    {
        get 
        { 
            return m_instance; 
        }
    }

    private void Awake()
    {
        if (m_instance == null)
        {
            m_instance = this;
        }
        else
        {
            Destroy(this.gameObject);
        }
    }

    [ContextMenu("Start Microphone")]
    public void StartMicrophone()
    {
        string microphoneName = GetMicrophoneDeviceName();
        m_audioClip = Microphone.Start(microphoneName, true, m_audioClipLength, m_samplingRate);
        string clipId = m_audioClip.GetInstanceID().ToString();
        m_audioClip.name = "GGMicAudioClip_" + clipId;
    }

    [ContextMenu("Stop Microphone")]
    public void StopMicrophone()
    {
        string microphoneName = GetMicrophoneDeviceName();
        Microphone.End(microphoneName);
    }

    public bool IsRecording()
    {
        string microphoneName = GetMicrophoneDeviceName();
        return Microphone.IsRecording(microphoneName);
    }

    public int GetPosition()
    {
        string microphoneName = GetMicrophoneDeviceName();
        return Microphone.GetPosition(microphoneName);
    }

    public string GetMicrophoneDeviceName()
    {
        return Microphone.devices[0];
    }

    public AudioClip GetMicrophoneAudioClip()
    {
        return m_audioClip;
    }

    private void Start()
    {
        StartMicrophone();
        SceneManager.sceneLoaded += OnSceneLoaded;
    }

    void OnSceneLoaded(Scene scene, LoadSceneMode mode)
    {
        StartMicrophone();
    }

    private void Update()
    {
        if (m_debugMicrophone == true)
        {
            if ((m_audioClip != null))
            {
                float micInputVolume = GetVolumeFromMicrophone() * m_microphoneSensitivity;
                if (micInputVolume < m_threshold)
                {
                    micInputVolume = 0;
                }
                m_micInputVolume = micInputVolume;
                m_audioClipId = m_audioClip.GetInstanceID();
            }
            m_microphonePosition = GetPosition();
        }
    }

    public float GetVolumeFromMicrophone()
    {
        return GetVolumeFromAudioClip(Microphone.GetPosition(Microphone.devices[0]), m_audioClip);
    }

    float GetVolumeFromAudioClip(int clipPosition, AudioClip audioClip)
    {
        if (clipPosition - m_sampleWindow < 0)
        {
            return 0;
        }

        float[] data = new float[m_sampleWindow];
        audioClip.GetData(data, clipPosition - m_sampleWindow);

        float totalVolume = 0;
        for (int i = 0; i < data.Length; i++)
        {
            totalVolume += Mathf.Abs(data[i]);
        }

        float averageVolume = totalVolume / m_sampleWindow;
        return averageVolume;
    }

    public void EnableMicrophoneDebugger(bool enabled)
    {
        m_debugMicrophone = enabled;
    }

    public void UpdateAudioClipLength(float clipLength)
    {
        m_audioClipLength = (int)clipLength;
    }

    public void SetSamplingRate(int samplingRate)
    {
        m_samplingRate = samplingRate;
    }
}

[olaysia]

Here's my custom photon recorder input. I basically copied it off Photon's MicWrapper script except instead of calling Microphone.Start(), which steals the mic input from the lip sync, I simply get the audio clip from my central microphone with the call GGMicrophone.Instance.GetMicrophoneAudioClip()

using UnityEngine;
using Photon.Voice;

public class GGMicrophoneInputForPhoton : IAudioReader<float>
{
    public GGMicrophoneInputForPhoton()
    {
        GGMicrophone.Instance.StartMicrophone();
    }

    public int SamplingRate
    {
        get
        {
            AudioClip clip = GGMicrophone.Instance.GetMicrophoneAudioClip();
            return clip.frequency;
        }
    }

    public int Channels
    {
        get
        {
            AudioClip clip = GGMicrophone.Instance.GetMicrophoneAudioClip();
            return clip.channels;
        }
    }

    public string Error { get; private set; }

    public void Dispose()
    {
        GGMicrophone.Instance.StopMicrophone();
    }

    private int micPrevPos;
    private int micLoopCnt;
    private int readAbsPos;

    public bool Read(float[] buffer)
    {
        if (Error != null)
        {
            return false;
        }
        int micPos = GGMicrophone.Instance.GetPosition();
        // loop detection
        if (micPos < micPrevPos)
        {
            micLoopCnt++;
        }
        micPrevPos = micPos;

        AudioClip clip = GGMicrophone.Instance.GetMicrophoneAudioClip();

        var micAbsPos = micLoopCnt * clip.samples + micPos;

        if (clip.channels == 0)
        {
            Error = "Number of channels is 0 in Read()";
            //logger.LogError("[PV] MicWrapper: " + Error);
            return false;
        }
        var bufferSamplesCount = buffer.Length / clip.channels;

        var nextReadPos = this.readAbsPos + bufferSamplesCount;
        if (nextReadPos < micAbsPos)
        {
            clip.GetData(buffer, this.readAbsPos % clip.samples);
            this.readAbsPos = nextReadPos;
            return true;
        }
        else
        {
            return false;
        }
    }
}

To initialise this custom input I simply pop this script on a GameObject in my scene:

public class GGMicrophoneInputForPhotonInitializer : MonoBehaviour
{
    private void Start()
    {
        Recorder recorder = PhotonVoiceNetwork.Instance.PrimaryRecorder;
        if (recorder != null)
        {
            recorder.SourceType = Recorder.InputSourceType.Factory;
            recorder.InputFactory = () => new GGMicrophoneInputForPhoton();
        }
        else
        {
            Debug.LogError("Could not set recorder's input source type because no recorder was found.");
        }
    }
}

And here is my central Microphone:

public class GGMicrophone : MonoBehaviour
{
    [SerializeField] AudioClip m_audioClip;
    [Tooltip("Enable to see the Mic Input Volume")][SerializeField] bool m_debugMicrophone = true;
    [Range(0f, 1f)] [SerializeField] float m_micInputVolume = 0f;
    [Range(0, 1000000)] [SerializeField] int m_microphonePosition;
    [SerializeField] int m_audioClipId;
    [SerializeField] float m_microphoneSensitivity = 50f;
    [SerializeField] float m_threshold = 0.1f;
    int m_samplingRate = 16000;
    int m_sampleWindow = 64;
    int m_audioClipLength = 1;

    static private GGMicrophone m_instance;
    static public GGMicrophone Instance
    {
        get 
        { 
            return m_instance; 
        }
    }

    private void Awake()
    {
        if (m_instance == null)
        {
            m_instance = this;
        }
        else
        {
            Destroy(this.gameObject);
        }
    }

    [ContextMenu("Start Microphone")]
    public void StartMicrophone()
    {
        string microphoneName = GetMicrophoneDeviceName();
        m_audioClip = Microphone.Start(microphoneName, true, m_audioClipLength, m_samplingRate);
        string clipId = m_audioClip.GetInstanceID().ToString();
        m_audioClip.name = "GGMicAudioClip_" + clipId;
    }

    [ContextMenu("Stop Microphone")]
    public void StopMicrophone()
    {
        string microphoneName = GetMicrophoneDeviceName();
        Microphone.End(microphoneName);
    }

    public bool IsRecording()
    {
        string microphoneName = GetMicrophoneDeviceName();
        return Microphone.IsRecording(microphoneName);
    }

    public int GetPosition()
    {
        string microphoneName = GetMicrophoneDeviceName();
        return Microphone.GetPosition(microphoneName);
    }

    public string GetMicrophoneDeviceName()
    {
        return Microphone.devices[0];
    }

    public AudioClip GetMicrophoneAudioClip()
    {
        return m_audioClip;
    }

    private void Start()
    {
        StartMicrophone();
        SceneManager.sceneLoaded += OnSceneLoaded;
    }

    void OnSceneLoaded(Scene scene, LoadSceneMode mode)
    {
        StartMicrophone();
    }

    private void Update()
    {
        if (m_debugMicrophone == true)
        {
            if ((m_audioClip != null))
            {
                float micInputVolume = GetVolumeFromMicrophone() * m_microphoneSensitivity;
                if (micInputVolume < m_threshold)
                {
                    micInputVolume = 0;
                }
                m_micInputVolume = micInputVolume;
                m_audioClipId = m_audioClip.GetInstanceID();
            }
            m_microphonePosition = GetPosition();
        }
    }

    public float GetVolumeFromMicrophone()
    {
        return GetVolumeFromAudioClip(Microphone.GetPosition(Microphone.devices[0]), m_audioClip);
    }

    float GetVolumeFromAudioClip(int clipPosition, AudioClip audioClip)
    {
        if (clipPosition - m_sampleWindow < 0)
        {
            return 0;
        }

        float[] data = new float[m_sampleWindow];
        audioClip.GetData(data, clipPosition - m_sampleWindow);

        float totalVolume = 0;
        for (int i = 0; i < data.Length; i++)
        {
            totalVolume += Mathf.Abs(data[i]);
        }

        float averageVolume = totalVolume / m_sampleWindow;
        return averageVolume;
    }

    public void EnableMicrophoneDebugger(bool enabled)
    {
        m_debugMicrophone = enabled;
    }

    public void UpdateAudioClipLength(float clipLength)
    {
        m_audioClipLength = (int)clipLength;
    }

    public void SetSamplingRate(int samplingRate)
    {
        m_samplingRate = samplingRate;
    }
}

[ErickMarinC]

Thank you so much! 

[violinviolin19]

Hi olaysia, thanks so much for the information! This has really helped my project. Would you also be able to share the lip sync clip updater? I am not able to get mine to work properly.