Forum Discussion
Oculus Quest + Intel RealSense D435i + Unity
Hello everybody,
I'm designing a VR performance, in which the user would interact with a real-time point cloud of the space around them.
I am using an Intel RealSense D435i depth camera (all the processing is done in the device) and an Oculus Quest, and I'm developing in Unity.
My question is:
Is it technically possible for the Quest to communicate with the RealSense camera, through the USB-C port ?
If not, is it a hardware limitation (for example, the Quest can't even power the RealSense) or is it a driver / android library / unity sdk limitation ?
So far, my test pointcloud-viewer project builds without errors and when I connect the Realsense to the Quest I get the familiar sound of the Quest recognizing that something connected to it, but the actual app does not show any pointcloud at all, and the depth camera is clearly inactive (no flashing laser, no IR illumination in the room when viewed in Passthrough mode).
This is new ground, so any help would be greatly appreciated!
I'm designing a VR performance, in which the user would interact with a real-time point cloud of the space around them.
I am using an Intel RealSense D435i depth camera (all the processing is done in the device) and an Oculus Quest, and I'm developing in Unity.
My question is:
Is it technically possible for the Quest to communicate with the RealSense camera, through the USB-C port ?
If not, is it a hardware limitation (for example, the Quest can't even power the RealSense) or is it a driver / android library / unity sdk limitation ?
So far, my test pointcloud-viewer project builds without errors and when I connect the Realsense to the Quest I get the familiar sound of the Quest recognizing that something connected to it, but the actual app does not show any pointcloud at all, and the depth camera is clearly inactive (no flashing laser, no IR illumination in the room when viewed in Passthrough mode).
This is new ground, so any help would be greatly appreciated!
9 Replies
- Have you connected to the device with adb logcat to see what the error messages say, if there are any?(https://developer.oculus.com/documentation/mobilesdk/latest/concepts/mobile-adb/)I could imagine that Oculus configured the version of Android that they're running to not grant permission for any camera to user apps, because they don't want people to br able to access the integrated ones (at least for now). But that's just a guess.Which Unity SDK are you using? This one? https://github.com/IntelRealSense/librealsense/tree/master/wrappers/unity
I just
checked out the latest release of the Unity plugin found here:In case
that is what you're using, I'm afraid it won't work like that. The unitypackage
contains a C# wrapper (Intel.RealSense.dll) and the actual native dll
(realsense2.dll). The latter one is for Windows only if I'm not mistaken. The
wrapper that is simply a .NET library, will compile fine for Android as you
say. However, it will try to load the native dll at runtime, which fails since
it wasn't compiled for Android.In order
to get a native Android library (.aar) that you can use in Unity with the C#
wrapper, you might need to compile the native android library yourself as
outlined here: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/android/readme.mdAlternatively
you could try downloading the pre-built android library here: https://dl.bintray.com/intel-realsense/librealsense/com/intel/realsense/librealsense/2.22.0/Throw
that into Unity and see if it gets you anywhere. Again, looking at logs with
adb logcat will be helpful here.
JulienK said:
I just
checked out the latest release of the Unity plugin found here:In case
that is what you're using, I'm afraid it won't work like that. The unitypackage
contains a C# wrapper (Intel.RealSense.dll) and the actual native dll
(realsense2.dll). The latter one is for Windows only if I'm not mistaken. The
wrapper that is simply a .NET library, will compile fine for Android as you
say. However, it will try to load the native dll at runtime, which fails since
it wasn't compiled for Android.In order
to get a native Android library (.aar) that you can use in Unity with the C#
wrapper, you might need to compile the native android library yourself as
outlined here: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/android/readme.mdAlternatively
you could try downloading the pre-built android library here: https://dl.bintray.com/intel-realsense/librealsense/com/intel/realsense/librealsense/2.22.0/Throw
that into Unity and see if it gets you anywhere. Again, looking at logs with
adb logcat will be helpful here.
Thanks for your reply and your time JulienK !
I reached to the exact same realization, minutes after posting the original question, and went on to build a native Android library from source, from the windows command line. The build was succesful and I placed the generated .aar file in the Plugins folder, alongside Intel.RealSense.dll and realsense2.dll, but the results were the same.
No issues in Unity, no pointcloud in the Quest.
I will definitely use the logcat tomorrow, and post any updates.
The thought that Oculus might have disabled access to anything appearing as a camera on Android, is frightening...- Interesting results from adb logcat (relevant part of the log attached below):
Lines 6-8:
E Unity : Unable to find libAudioPluginOculusSpatializer
E Unity : Unable to find librealsense
E Unity : Unable to find realsense
and later on (Lines 23-28):
MarshalDirectiveException: Cannot marshal type 'System.Object'.
at Intel.RealSense.NativeMethods.rs2_create_frame_queue
at Intel.RealSense.FrameQueue.set_Capacity
at Intel.RealSense.NativeMethods.rs2_create_processing_block_fptr
at Intel.RealSense.FrameQueue..ctor
at RsProcessingPipe.Awake ()
and finally (Lines 32-41):
MarshalDirectiveException: Cannot marshal type 'System.Object'.
at Intel.RealSense.NativeMethods.rs2_get_api_version
at Intel.RealSense.NativeMethods.rs2_get_api_version
at Intel.RealSense.Pipeline.Create ()
at Intel.RealSense.Pipeline..ctor ()
at RsDevice.OnEnable ()
Rethrow as TypeInitializationException: The type initializer for 'Intel.RealSense.Context' threw an exception.
at Intel.RealSense.Pipeline.Create ()
at Intel.RealSense.Pipeline..ctor ()
at RsDevice.OnEnable () [0x00000] - Just to make sure:Did you tick the Android checkbox in the Import Settings of the library?
- Yes, I actually had all three platforms (Editor, Standalone, Android) checked for all the libraries. Whether the "Load on startup" was checked or not, made no difference.
- I am happy to report that thanks to great developer support from Intel, I got the Oculus Quest - Intel Realsense integration working!The Quest is perfectly able to drive an Intel RealSense D435i depth camera connected through the USB-C port, and access the Point Cloud, Depth and RGB streams.For future reference, here is the conversation on GitHub leading to the solution: https://github.com/IntelRealSense/librealsense/issues/4155I will update this thread with a detailed walkthrough.
- Hey George! I'm working on a quite similar usecase and would be super thankful if you could elaborate your workaround a bit more in depth (although it's been a while) Is there any repo available for this? Thank you very much in advance!
I have the D435 on the robot, and want to stream the video to the Oculus. This shouldn't be too hard!
