Skip to main content
georgeadamon
New Participant
June 4, 2019

Oculus Quest + Intel RealSense D435i + Unity

Hello everybody, 

I'm designing a VR performance, in which the user would interact with a real-time point cloud of the space around them.
I am using an Intel RealSense D435i depth camera (all the processing is done in the device) and an Oculus Quest, and I'm developing in Unity.

My question is:
Is it technically possible for the Quest to communicate with the RealSense camera, through the USB-C port ?
If not, is it a hardware limitation (for example, the Quest can't even power the RealSense) or is it a driver / android library / unity sdk limitation ?

So far, my test pointcloud-viewer project builds without errors and when I connect the Realsense to the Quest I get the familiar sound of the Quest recognizing that something connected to it, but the actual app does not show any pointcloud at all, and the depth camera is clearly inactive (no flashing laser, no IR illumination in the room when viewed in Passthrough mode).

This is new ground, so any help would be greatly appreciated!

    9 replies

    julienkay
    Honored Contributor
    June 4, 2019
    Have you connected to the device with adb logcat to see what the error messages say, if there are any?
    (https://developer.oculus.com/documentation/mobilesdk/latest/concepts/mobile-adb/)

    I could imagine that Oculus configured the version of Android that they're running to not grant permission for any camera to user apps, because they don't want people to br able to access the integrated ones (at least for now). But that's just a guess.


    julienkay
    Honored Contributor
    June 4, 2019

    I just
    checked out the latest release of the Unity plugin found here:



    https://github.com/IntelRealSense/librealsense/tree/development/wrappers/unity/Assets/RealSenseSDK2.0



    In case
    that is what you're using, I'm afraid it won't work like that. The unitypackage
    contains a C# wrapper (Intel.RealSense.dll) and the actual native dll
    (realsense2.dll). The latter one is for Windows only if I'm not mistaken. The
    wrapper that is simply a .NET library, will compile fine for Android as you
    say. However, it will try to load the native dll at runtime, which fails since
    it wasn't compiled for Android.



     



    In order
    to get a native Android library (.aar) that you can use in Unity with the C#
    wrapper, you might need to compile the native android library yourself as
    outlined here: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/android/readme.md





    Alternatively
    you could try downloading the pre-built android library here: https://dl.bintray.com/intel-realsense/librealsense/com/intel/realsense/librealsense/2.22.0/



    Throw
    that into Unity and see if it gets you anywhere. Again, looking at logs with
    adb logcat will be helpful here.




    georgeadamonAuthor
    New Participant
    June 5, 2019

    JulienK said:

    I just
    checked out the latest release of the Unity plugin found here:



    https://github.com/IntelRealSense/librealsense/tree/development/wrappers/unity/Assets/RealSenseSDK2.0



    In case
    that is what you're using, I'm afraid it won't work like that. The unitypackage
    contains a C# wrapper (Intel.RealSense.dll) and the actual native dll
    (realsense2.dll). The latter one is for Windows only if I'm not mistaken. The
    wrapper that is simply a .NET library, will compile fine for Android as you
    say. However, it will try to load the native dll at runtime, which fails since
    it wasn't compiled for Android.



     



    In order
    to get a native Android library (.aar) that you can use in Unity with the C#
    wrapper, you might need to compile the native android library yourself as
    outlined here: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/android/readme.md





    Alternatively
    you could try downloading the pre-built android library here: https://dl.bintray.com/intel-realsense/librealsense/com/intel/realsense/librealsense/2.22.0/



    Throw
    that into Unity and see if it gets you anywhere. Again, looking at logs with
    adb logcat will be helpful here.






    Thanks for your reply and your time JulienK !

    I reached to the exact same realization, minutes after posting the original question, and went on to build a native Android library from source, from the windows command line. The build was succesful and I placed the generated .aar file in the Plugins folder, alongside Intel.RealSense.dll and realsense2.dll, but the results were the same.
    No issues in Unity, no pointcloud in the Quest.

    I will definitely use the logcat tomorrow, and post any updates.

    The thought that Oculus might have disabled access to anything appearing as a camera on Android, is frightening... 
    georgeadamonAuthor
    New Participant
    June 5, 2019
    Interesting results from adb logcat (relevant part of the log attached below):


    Lines 6-8:
    E Unity   : Unable to find libAudioPluginOculusSpatializer
    E Unity   : Unable to find librealsense
    E Unity   : Unable to find realsense

    and later on (Lines 23-28):
    MarshalDirectiveException: Cannot marshal type 'System.Object'.
        at Intel.RealSense.NativeMethods.rs2_create_frame_queue
        at Intel.RealSense.FrameQueue.set_Capacity
        at Intel.RealSense.NativeMethods.rs2_create_processing_block_fptr
        at Intel.RealSense.FrameQueue..ctor
        at RsProcessingPipe.Awake ()

    and finally (Lines 32-41):
    MarshalDirectiveException: Cannot marshal type 'System.Object'.
       at Intel.RealSense.NativeMethods.rs2_get_api_version
       at Intel.RealSense.NativeMethods.rs2_get_api_version 
       at Intel.RealSense.Pipeline.Create ()
       at Intel.RealSense.Pipeline..ctor () 
       at RsDevice.OnEnable ()
     Rethrow as TypeInitializationException: The type initializer for 'Intel.RealSense.Context' threw an exception.
       at Intel.RealSense.Pipeline.Create ()
       at Intel.RealSense.Pipeline..ctor ()
       at RsDevice.OnEnable () [0x00000] 
    julienkay
    Honored Contributor
    June 5, 2019
    Just to make sure:
    Did you tick the Android checkbox in the Import Settings of the library?
    georgeadamonAuthor
    New Participant
    June 5, 2019
    Yes, I actually had all three platforms (Editor, Standalone, Android) checked for all the libraries. Whether the "Load on startup" was checked or not, made no difference.
    georgeadamonAuthor
    New Participant
    June 18, 2019
    I am happy to report that thanks to great developer support from Intel, I got the Oculus Quest - Intel Realsense integration working!

    The Quest is perfectly able to drive an Intel RealSense D435i depth camera connected through the USB-C port, and access the Point Cloud, Depth and RGB streams.

    For future reference, here is the conversation on GitHub leading to the solution: https://github.com/IntelRealSense/librealsense/issues/4155

    I will update this thread with a detailed walkthrough.

      Frankfurt
      New Participant
      November 16, 2020
      Hey George! I'm working on a quite similar usecase and would be super thankful if you could elaborate your workaround a bit more in depth (although it's been a while) Is there any repo available for this? Thank you very much in advance!
        user848199168448882
        New Participant
        October 27, 2021

        I have the D435 on the robot, and want to stream the video to the Oculus. This shouldn't be too hard!

         

        Terms & ConditionsAccessibility statement