Data Streaming from Unity VR app to mobile (iOS/android/web) client

The peer-to-peer networking looked really promising, and it would be easy to pick up. I got lost trying to figure out what a peerID is, and whether or not I could create a peerID through a mobile app.

I've abandoned trying to do it with Unity's networking. I can make TCP connections between a react-native mobile and a unity build for the headset. I had to hack it together by logging the IP address of my headset and entering it into the mobile app. I'm struggling to find a good way to discover devices on a network. It would be awesome if I could use iOS's Bonjour or Androids NSD, but I'm kinda blocked on how I might run the Java code required for Androids NSD because I couldn't find any C# libraries for it.

I made an algorithm that discovers devices on a local network that are listening to a certain port. I got the IP address and Subnet from Node.js (os module). Then, used npm's NetMask to find the network address and I just iterated through every available address attempting to make connections. The next step would've been sending a handshake, so I can tell if the listener was a device that I programmed.

This is a bad system though because you're not guaranteed to be allowed to listen on the same port every time the app runs, and I worry about how flexible this system is on networks with different configurations.

It depends a bit on what your end goal is. For you, it could be as simple as just manually entering in the IP address every time you need the devices to connect, or you could configure your router to give those devices static IP addresses.

The TCP connection is as simple as a server opening a port, and a client making a connection to the server's ip address and port.

My final iteration of device discovery over TCP was using Unity's old network discovery (their new networking library has some sample code for network discovery, but it wasn't plug and play like this older one is)

The app on my headset made the device a server, so it opened a listener on a port. I used the network discovery to broadcast/multicast my ip address and port. My tablet/pc app would then listen for the broadcast and the UI had a list of "discovered devices" to connect to.

Some issues with this are that Oculus' VRC checks, their submission guidelines, do not allow changing wifi state, and the network discovery requires CHANGE_WIFI_MULTICAST_STATE, so publicly providing the app on their app store would be met with resistance.

I'm now using a Node.js server on aws with Websockets instead! It's more similar to Facebooks own relay servers.

I think their example would frequently block in Unity though. Causing your headset/desktop app to freeze. I'll share some bits of my code. It's a grouping of a few different solutions from online, and I can't find them anymore. This is contrived and shortened from my code because it contains another layer of command encapsulation that would be too much to breakdown in a few minutes:

using System.Collections;
using System.Collections.Generic;
using System;
using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Text;
using UnityEngine;

public class CommandServer : MonoBehaviour
{
    private Int32 port; // The port assigned to this listener by the OS.

    private IPAddress IPV4Address;

    TcpListener server;

    private bool isListening; // true if CommandServer is currently listening on a port.

    private const float LISTEN_INTERVAL = .3f; // this will be a bottlneck if we get more than 3 commands a second.



    void Start()
    {
        StartListener();
        StartCoroutine(Listen());
    }



    private void StartListener()
    {
        string ipv4 = IPManager.GetIP(ADDRESSFAM.IPv4);
        try
        {
            IPAddress localAddr = IPAddress.Parse(ipv4);

            // TcpListener server = new TcpListener(port);
            server = new TcpListener(localAddr, 0); // 0 for port means OS will assign port for us

            // Start listening for client requests.
            server.Start();
            isListening = true;
            setPort(server); // make new port publicly accessible, so it can be included in broadcasts.
            setIPV4Address(localAddr); // making new IPV4 address visible for the same reason.
            Debug.Log("Binded to Port: " + port);
            Debug.Log("At IPV4: " + IPV4Address);
        }
        catch (FormatException e) // will happen if the IP is invalid.
        {
            Debug.LogException(e);
            Debug.Log("IPAddress: " + ipv4);
        }
    }



    private IEnumerator Listen()
    {
        while (true)
        {
            try
            {
                // Waiting for a connection
                if (server.Pending()) // this is so we only AcceptTcpClient if there is one ready
                {
                    TcpClient client = server.AcceptTcpClient();
                    //StartCoroutine(ExchangeCommands(client)); // my protocol was to read one command then send a response.
                }
            }
            catch (SocketException e)
            {
                Debug.Log("SocketException: " + e);
                StopListener();
                StartListener();
            }
            catch (System.InvalidOperationException e) // will happen if we call server.Pending() before server is properly started i.e. StartListener fails.
            {
                Debug.Log("InvalidOperationException: " + e);
                StopListener();
                StartListener();
            }
            catch (NullReferenceException e) // I've seen this happen once, and it was because the server somehow got set to null
            {
                StopListener();
                StartListener();
            }
            yield return new WaitForSecondsRealtime(LISTEN_INTERVAL); // look for a new connection every LISTEN_INTERVAL seconds.
        }
    }

    private void StopListener()
    {
        try
        {
            server.Stop();
            isListening = false;
        }
        catch (SocketException e)
        {
            Debug.Log("SocketException: " + e);
        }
        catch (NullReferenceException e) // server was never successfully initialized
        {
            Debug.LogException(e);
        }
    }


    public Int32 getPort()
    {
        return port;
    }

    public string getIPV4Address()
    {
        return IPV4Address.ToString();
    }

    private void setIPV4Address(IPAddress IPV4Address)
    {
        this.IPV4Address = IPV4Address; // external objects cannot change value of private IPV4Address, so give them a string.
    }

    private void setPort(TcpListener server)
    {
        IPEndPoint localEndPoint = (IPEndPoint)server.LocalEndpoint;
        port = localEndPoint.Port;
    }

    public bool getIsListening()
    {
        return isListening;
    }

    private void OnDestroy()
    {
        StopListener();
    }
}

Now this example doesn't bind to one specific port. You're not technically guaranteed to have access to the same port every time, so my server would bind to any available port and broadcast its signature using UNet Network Discovery. You'll need the ip address and the port this listener binds too. I've made it log to the console, so you can manually enter it into your python app to start. 

Edit: Added Using Statements

Yes, good detective work. I forgot that wasn't a native library sorry! I changed one thing. I can't remember what conditions caused this, but periodically it would return a loopback address, and this won't work if you're trying to do inter computer communication. I just added one line under: 

if ((item.NetworkInterfaceType == _type1 || item.NetworkInterfaceType == _type2) && item.OperationalStatus == OperationalStatus.Up)

outside of the #endif

using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;

public class IPManager
{
    // loopback interfaces are skipped.
    // returns the interface belonging to IPV4, if multiple interfaces exist, the last interface found gets returned.
    public static string GetIP(ADDRESSFAM Addfam)
    {
        //Return null if ADDRESSFAM is Ipv6 but Os does not support it
        if (Addfam == ADDRESSFAM.IPv6 && !Socket.OSSupportsIPv6)
        {
            return null;
        }

        string output = "";

        foreach (NetworkInterface item in NetworkInterface.GetAllNetworkInterfaces())
        {
#if UNITY_EDITOR_WIN || UNITY_STANDALONE_WIN
            NetworkInterfaceType _type1 = NetworkInterfaceType.Wireless80211;
            NetworkInterfaceType _type2 = NetworkInterfaceType.Ethernet;

            // test for wireless or ethernet
            if ((item.NetworkInterfaceType == _type1 || item.NetworkInterfaceType == _type2)
                && item.OperationalStatus == OperationalStatus.Up)
#endif
                if (item.NetworkInterfaceType != NetworkInterfaceType.Loopback) // skip loopback
                {
                    {
                        foreach (UnicastIPAddressInformation
                            ip
                            in
                            item.GetIPProperties().UnicastAddresses
                        )
                        {
                            //IPv4
                            if (Addfam == ADDRESSFAM.IPv4)
                            {
                                if (
                                    ip.Address.AddressFamily ==
                                    AddressFamily.InterNetwork
                                )
                                {
                                    output = ip.Address.ToString();
                                }
                            } //IPv6
                            else if (Addfam == ADDRESSFAM.IPv6)
                            {
                                if (
                                    ip.Address.AddressFamily ==
                                    AddressFamily.InterNetworkV6
                                )
                                {
                                    output = ip.Address.ToString();
                                }
                            }
                        }
                    }
                }
        }
        return output;
    }
}

public enum ADDRESSFAM
{
    IPv4,
    IPv6
}

Please excuse the formatting. I was using vscode's 'prettier' before I realized the path of least resistance is visual studio.

Just for reference, my protocol is always a connection from the client to the headset. They both exchange one message. The client sends a message first, and upon receiving and acting on that message, the headset/listener sends a response. To read the clients message, my function stack eventually comes to this:

private const string END_OF_STREAM_STRING = "\n\n\nEOF_END_EOF_END\n\n\n";
private byte[] END_OF_STREAM;    

public static InterdeviceCommand readCommandFromStream(System.Net.Sockets.NetworkStream networkStream)
    {
        byte[] buffer = new byte[8192];
        int bytesRead;
        string decodedRead = "";
        networkStream.ReadTimeout = READ_TIMEOUT;

        // read stream until there is nothing left to read.
        while ((bytesRead = networkStream.Read(buffer, 0, buffer.Length)) > 0)
        {
            decodedRead += Encoding.ASCII.GetString(buffer, 0, bytesRead); // This only decodes correctly if ASCII chars are 1 byte. Otherwise, some chars will be half read from the stream.
            if (decodedRead.Contains(END_OF_STREAM_STRING))
            {
                string jsonData= getDataBeforeEndOfStream(decodedRead); // cut everything from the END_OF_STREAM sequence onward. We only deserialize what comes before.
                return JsonUtility.FromJson<InterdeviceCommand>(jsonData);
            }
        }
        throw new EndOfCommandNotFound("END_OF_STREAM not found, so command could not be deserialized safely");
    }


    private static string getDataBeforeEndOfStream(string data)
    {
        string result;
        int indexOfEndOfStream;
        indexOfEndOfStream = data.IndexOf(END_OF_STREAM_STRING);
        if (indexOfEndOfStream >= 0)
            result = data.Substring(0, indexOfEndOfStream); // remove END_OF_STREAM, and everthing after it, from the data.
        else // we shouldn't ever execute this else based on how I intend to call this message. Just incase the code changes
        {
            throw new EndOfCommandNotFound("END_OF_STREAM not found, so command could not be deserialized safely");
        }
        return result;
    }

I'd recommend using it as a guide rather than a full plug and play solution. There's stuff like custom exceptions and Data Types that I'm omitting for brevity. It won't be too much an undertaking to try and get your code to a point where the client sends a string and the listener responds with another string. You should try and get that working before you try to fully mimic this code.

My server and client have a shared data type, InterdeviceCommand. Each end creates a command, serializes it into json, then sends it with

        string jsonData = JsonUtility.ToJson(this);
        if (jsonData.Contains(END_OF_STREAM_STRING))
        {
            throw new SerializedObjectContainsEOF(string.Format("We can't Stream.Write() a serialized object-containing-EOF, our communication protocol will desync. Attempted to Stream.Write() this object: {0}", jsonData));
        }
        byte[] bytes = Encoding.ASCII.GetBytes(jsonData);
        networkStream.WriteTimeout = WRITE_TIMEOUT;
        networkStream.Write(bytes, 0, bytes.Length); // todo, make this asynchronous
        networkStream.Write(END_OF_STREAM, 0, END_OF_STREAM.Length);

I'm also ending each stream with an END_OF_STREAM string. I needed that string, to tell that a client has finished it's message. After reading a message, it's deserialized into that original class.

You're mostly working with the NetworkStream class in unity, and that's returned by 

TcpClient client = server.AcceptTcpClient();
NetworkStream networkStream = client.GetStream();

First step is just sending a string from the python client, reading it on the server, and Debug.Log() it. I wouldn't worry about anything else.

To answer your second question. This is where it gets tricky. You can't really assign the TCPListener an IP address. You can only listen on an IP that has been assigned to your system by your local router. IPManager is 'finding' the first available IP address that we have been assigned, and can listen on.

If your router assigns IP addresses dynamically, you won't be able to use the same IP addresses on every execution. You could make your router assign IP addresses statically, so every computer on your system will always be assigned the same IP address. A Static IP address isn't the cleanest solution, and depending on your networking experience, it might be easier to do one of the next two options.

Start with option 1, just to try and get it working. If you do, option 2 is another more robust step.

Option 1: Display that assigned port and ip address in a UI. When you launch your app, you can navigate to this UI, record the IP address and port. Then, you can manually enter it into your python app.

Option 2: Device Discovery

The most simple form of this is constantly broadcasting your registered port and ip address to a hardcoded port.

1. your headset is assigned an ip address by the router, and it binds to an arbitrary port assigned by the OS.

2. You headset app obtains the Broadcast IP address from the assigned IP address 

3. Your headset app packages this information into a 'BroadcastMessage'

4. Your headset app serializes the BroadcastMessage and Broadcasts it

5. Your client listens for broadcasts on a shared port, shared between the listener and the client app.

6. Your client receives the broadcast and parses it into "BroadcastMessage'

7. Your client reads values from BroadcastMessage i.e. BroadcastMesssage.IP and BroadcastMessage.Port

8. Using those values, your python client can get a connection to your headset app without you having to constantly input the port and address. 

UNet discovery kind of does this for you. It's going to be more sophisticated. I made a class that inherits NetworkDiscovery, and I changed the BroadcastData to be something that contains my IP address and Port.