Skip to main content
peterept
Up and Coming
February 27, 2016

OVR LipSync Bug: Multiple targets mouth the same?

Hi,

I have a scene with multiple users using OVR Lip Sync.

However, if one user talks *all* users mouth movements are sync'd to that (on their local game).

If I debug out OVRLipSyncContext, they each have a unique context id, so it should keep them separated during decoding. however, it doesn't appear to work internally.

It is easy to repro using the OVR Lip Sync install:

1. Create a new scene
2. Drag in a "LipSyncInterface" prefab
3. Drag in "LipSyncMorphTarget_RobotTexture" prefab
3.1 Name it "Audio File", and delete the OVRLipSyncMicInput, enable OVRLipSyncTestAudio, drag on "OVRLipSync/Audio/vox_lp_01" onto the audio source
4. Drag in a second "LipSyncMorphTarget_RobotTexture" prefab
4.1 rename it "Mic" . Duplicate the material in OVRLipSyncContextTextureFlip so it is different.
Press PLAY
Expected:
The "Audio File' keeps mouthing that file while the "Mic" plays what I say
Actual:
The "Audio File" stops mouthing, and both mouth whatever I say.

I am on Unity 5.2.3p1 with OVR Lip Sync V1.0.1-beta.

Am I missing something obvious?
    This topic has been closed for replies.

    7 replies

    petereptAuthor
    Up and Coming
    February 27, 2016
    Ahh, found the issue.

    It is because the way OVRLipSyncContext works is it has a material for the mouth textures which has to match the material on the mouth renderer.

    Now, when you instantiate a prefab of it, obviously all objects have the same material so any change to one effects the other.

    What I failed to notice at the time was I didn't change both materials.

    The fix I used was to change the OVRLipSyncContext to not use the material but rather just access the instanced material on the mouth object with public Renderer MouthRenderer;

    MouthRenderer.material instead of material.

    Now it works great.
    Check out my Unity VR development blog: http://talesfromtherift.com @peterept
    petereptAuthor
    Up and Coming
    February 28, 2016
    If you are using Lip Sync with texture flip in a prefab, obviously you can edit OVRLipSyncContextTextureFlip.cs to access a new material. But if you want to avoid changing the Oculus scripts, here is a script that will do it for you. Place it on the same game object as the OVRLipSyncContextTextureFlip.

    It serves 2 purposes:

    1. Sets the material to point to the mouth material
    2. By accessing the Render.material at runtime it creates a new material instance so it is no longer shared between prefabs

    SeetupOVRLipSyncContextTextureFlip.cs:

    using UnityEngine;
    using System.Collections;

    [RequireComponent(typeof(OVRLipSyncContextTextureFlip))]
    public class SeetupOVRLipSyncContextTextureFlip : MonoBehaviour 
    {
    	public Renderer MouthRenderer;

    	void Awake() 
    	{
    		GetComponent<OVRLipSyncContextTextureFlip> ().material = MouthRenderer.material;
    	}
    }


    HTH someone else!

    I suggest in the future Oculus could change OVRLipSyncContextTextureFlip to take a reference to the renderer so it does this automatically as I expect most games are going to have this in a prefab.
    Check out my Unity VR development blog: http://talesfromtherift.com @peterept
      petergiokaris
      Inspiring
      February 29, 2016
      Thank you for finding this! We will fix our script to reference an instanced version of the material.
      Peter Giokaris Senior Software Engineer
      petergiokaris
      Inspiring
      February 29, 2016
      I have a fix that I will add the the OVRLipSyncTextureFlip script.

      You can add this bit of code at the end of the Start function, which will copy the material to a temp location,create a new material list and copy the temp back:

      // Instantiate the material that is set in this script to each renderer in the children
      Renderer [] renderers = GetComponentsInChildren<Renderer>();

      foreach (Renderer r in renderers)
      {
      if (r.sharedMaterial == material)
      {
      // Copy mat to temp location
      Material [] tmp = new Material[1];
      tmp [0] = new Material (r.sharedMaterial);

      // Create a new material list and re-copy back
      r.materials = new Material[1];
      r.materials [0].CopyPropertiesFromMaterial(tmp[0]);
      material = r.materials [0];
      }
      }

      This should fix the issue.
      Peter Giokaris Senior Software Engineer
      leftler
      Up and Coming
      March 1, 2016
      "petergiokaris" wrote:

      Material [] tmp = new Material[1];
      tmp [0] = new Material (r.sharedMaterial);
      Why is tmp an array of size 1 and not just?

      Material tmp = new Material (r.sharedMaterial);


      Was this just to match the style of the existing code or is there some kind of "this will create less garbage for the GC" optimisation I am not realizing that is important to do in Unity.
      petergiokaris
      Inspiring
      March 1, 2016
      This was to match the style of the existing code that I found from Unity forums.
      Peter Giokaris Senior Software Engineer
      petereptAuthor
      Up and Coming
      March 2, 2016
      Thanks Peter - I can confirm your fix works!

      In the context of a multi-user scenario, I assume it's quite a heavy operation to decode the audio stream, is there any way to decode the visemes only on the local users device, and then transmit them with voice data packets (to avoid all client devices decoding multiple audio streams) ?

      Peter
      Check out my Unity VR development blog: http://talesfromtherift.com @peterept
      Terms & ConditionsAccessibility statement