Skip to main content
Anonymous
June 11, 2019

On-device Speech Recognition on the Quest with Unity

Hi, I am working in Unity, and I have been trying to implement on-device speech recognition on the Oculus Quest. Knowing that the Quest uses Android, I tried using Android's speech API's through a plugin I found on the Unity asset store. The plugin gives me speech recognition on my Android Phone but not on the Oculus Quest. On the Quest I get an error saying that speech is not supported. Looking through the forums, I learned that the Oculus Go is a "bare bones" version of Android that doesn't have native Speech Recognition. I've been searching for other on-device speech recognition solutions, but so far it seems like most speech recognition solutions require the internet.
Can anyone recommend an on-device speech recognition solution that is compatible with the Quest, if one exists? Or will I have to go to an online-based solution?

      11 replies

      Anonymous
      June 13, 2019
      Bump
      ethanmh
      New Participant
      June 22, 2019
      I'm also very interested in finding an on-device speech recognition solution that is compatible with the Quest. Following.
      pdehaanST
      New Participant
      June 23, 2019
      I can verify that android builds of Pocketsphinx work on the quest. IME with Unity the Swig/JNI wrappers complicate things, you can just drop in the appropriate shared libraries and use C# interop (DllImport and such) to call into libraries straight from C# code.

      Still have the usual mobile device limitations (e.g. don't use massive vocabularies), but works fine otherwise.
      ethanmh
      New Participant
      June 28, 2019
      @pdehaanST How did you declare the functions to call? Did you have to use swift? Do you have an example of initializing I could see?
      pdehaanST
      New Participant
      June 28, 2019
      No swift involved for android platforms. Most of the work is handled by C# attributes from System.Runtime.InteropServices, for example:
      using System.RuntimeInteropServices;

      public static class SphinxAPI {
          [DllImport("pocketsphinx_jni")]
          public static extern IntPtr ps_args();

          [DllImport("pocketsphinx_jni")]
          public static extern IntPtr cmd_ln_init(IntPtr inout_cmdln, IntPtr defn, Int32 struct, IntPtr _nl);
      }

      ...declares functions that I can call from my C# code that are mapped to plain old C functions from the pocketsphinx_jni shared native library. So somewhere in a setup function I've written, I've got a line like:
      var config = SphinxAPI.cmd_ln_init(IntPtr.Zero, SphinxAPI.ps_args(), 1, IntPtr.Zero);

      IntPtr is basically just a C# wrapper around pointers for the C code. In the pocketsphinx library cmd_ln_init returns a cmd_ln_t* (see https://cmusphinx.github.io/doc/sphinxbase/cmd__ln_8h.html and https://cmusphinx.github.io/doc/sphinxbase/cmd__ln_8h.html as potential starting points for API docs), but my C# code doesn't know about that, I just need to store the pointer to pass into other functions.

      I've got a slightly more fleshed out example of a Unity project at https://github.com/pdehn/UnitySphinx, but that's currently Windows-only. You'll need a version of the libraries compiled for android to use with the quest (a quick solution is to just grab the .aar from the cmusphinx android demo repository at https://github.com/cmusphinx/pocketsphinx-android-demo/blob/master/aars/pocketsphinx-android-5prealpha-release.aar). You'd likely want to overhaul the code from my demo project as well. Of note, it runs all decoding in the Unity main thread (can lead to frame drops which is bad for VR) and you'd need to unpack all files that pocketsphinx reads (acoustic models, dictionary, language model, etc.) onto the filesystem, since Unity just packs everything into a .apk that sphinx wouldn't know how to read from.

      In theory you could also use the Java bindings generated using SWIG, but I've not touched that and can't really help there.
      Anonymous
      July 1, 2019
      @pdehaanST Thanks for the recommendation! I was able to get a working solution of PocketSphinx running on an Android phone using a project someone else made here. I won't be able to test it on the Quest until I get access to it next week. I'll post how it goes.

      Another solution recommended to me was to use TensorFlow Lite, but I haven't looked into it yet.
      checkbox
      Participant
      August 2, 2019
      Ahzee, can you post updates on how far you got? Really interested in this ?
      LeonKleinBretel
      Participant
      August 16, 2019
      Hi guys, any update on speech recognition? on-device speech recognition on the quest would be really useful.
      ognjentodic
      New Participant
      February 7, 2020
      My company (Keen Research) develops SDKs for on-device speech recognition for Android and iOS; we also offer Unity plugin. We license SDKs on commercial bases and offer a trial version of the SDK (works for 15min at the time) for evaluation purposes.

      Forum doesn't allow me to post links but you can use your favorite search engine and search for KeenASR.

      /Ogi
      NullLemon
      New Participant
      November 15, 2021

      Hi, Im currently trying out the KeenASR SDK but not not really straight forward in terms of debugging. Im getting "KeenASR SDK was not initialized properly" with no reason at all. By any chance do you have any advice?

      romeymerkel
      New Participant
      May 13, 2020
      Hi guys, is there any way to do this? Please help me with this. Been struggling to implement offline voice recognition on my Oculus quest. 
      Terms & ConditionsAccessibility statement