Mixed Reality Capture Tools (2.0) not working on Quest and test app crashes to ~1fps

[rib]

Hi, I'm trying to set up the Mixed Reality Capture Tools to be able to record a demo with the Oculus Quest. I've managed to calibrate a USB camera and uploaded the config onto the device under my app's files/ directory and I have OBS set up with the 'Oculus MRC' fg/bg sources as documented. I can connect to the device via OBS once I start my application but when I do I seem to just get one frame grabbed by OBS and the app framerate suddenly drops down to about one frame per second, with no visible updates in OBS beyond the first frame.

I wonder if anyone else had any luck using the latest Mixed Reality Capture tools (Oculus_MRC_Tools_2.0/1.41.0_RC1, OBS 23.2.1, Unity Integration Asset 1.40) and can maybe give some hints for what I might need to do to get it working?

[cly3d]

This video was uploaded today: https://www.youtube.com/watch?v=p1c8D_l0upI   
Hope its of help in some way.

[rib]

Thanks, it's a good overview of the technology and the setup process, though I think, afaikt, I did everything as described.

One thing that caused me some trouble for a while was that I was using OBS 24.x.x to start with and it turns out that the current plugin binaries are tied to the OBS 23.x.x ABI which wasn't immediately obvious.

I'll maybe take another stab and see if I have any luck.

[Lazius001]

Rib , i might be able to help but am stuck at the last hurdle, am not able to get my obs to connect the mrc.xml files.
1.Do you save the camera file to mrc.xml after doing a chroma or after finish calibration only during camera calibration(so before clicking chroma)?

 1b.I assume we can close the camera calibration window after saving the mrc.xml file?

2. To transfer mrc.xml file to quest, you put {youroculusdeveloper name}.{youroculusappnane}.files or .{youroculususername}.{beatsaber}.files. (example);

3. When I open OBS after putting all the plugging folders in the root, I click the + in the source, I see OculusMrcSource, do you see that too or do you see Oculus MRC? 

3b. The video capture source I assume was same as used during the camera calibration process, we don't use the screen (if left open) of the chroma from the canera calibration window from previous steps, correct? As mentioned I closed that window after saving the mrc.xml file as we are told to do chroma from obs, not calibration in previous steps, is that correct?

4, after i apply the filters pad/crop and change the relative settings, my fg abd bg dont change blue or green, so am stuck there,

4c, when i click connect after changing the ip in the OculusMrcSources, nothing happens.

I tried a lot of variations and would really appreciate your help on this. Please let me know if you able reply here or wish a Skype or messenger chat, or discord.

Thx 
Laz

[Lazius001]

Just update to say, all works, but just still enable to connect obs to the quest only... (Plugin says Oculus MRC and have the blue and green in green after activation of plugin).

[rib]

Hey, I haven't gotten around to trying again yet, but a few notes...

It looks like mrc.xml does store some chroma key info, so I suppose save as late as possible - though not sure why the device needs the chroma key of the camera?

Yeah, you have to close the calibration window so OBS can access your camera. Note: It will be important to ensure you configure your videos source / camera in OBS with the same resolution as was used in the calibration tool.

This is how I uploaded my calibration file:

adb shell mkdir -p /sdcard/Android/data/co.realfit.RealFit/files

adb push mrc.xml /sdcard/Android/data/co.realfit.RealFit/files

Theoretically for BeatSaber I think it would go here:

adb push mrc.xml /sdcard/Android/data/com.beatgames.beatsaber/files/

Out of curiousity I tried connecting to BeatSaber too but it seems like they don't support Oculus Quest mixed reality yet 😜 (couldn't connect) - hopefully they will support it in an update at some point.

In OBS the source I see is listed as 'Oculus MRC'. (Btw, I downgraded my OBS installation to 23.2.1 to get the plugin working for me).

Yeah the chroma keying is done in OBS (I don't really know what the purpose of the chroma key functionality in the calibration tool is).

For the fg/bg setup, can you maybe first confirm that if you look at the properties for the MRC sources then you see a wide rectangle that's blue on the left and green on the right? Essentially the cropping is cropping is cutting the left side of this source as the background and the right hand side as the foreground. If I just show the fg group I see green, and if I just show the bg group I see blue.

I guess the fact that it doesn't connect implies that either implies that your mrc.xml is not in the correct location for the application to pick up and enable MR connections, or perhaps that application doesn't yet support this feature (which I think is the case for BeatSaber currently). Have you perhaps tried testing this with a custom Unity project with the 1.40 Oculus Integration?

Hope some of those comments might possibly help. I'll try and take another stab myself at getting this working soon and try and share anything I learn.

Best of luck!

[Lazius001]

Hey Rip, thanks for your reply.

Yea I see fg/bg green/blue, all good there..i also downgraded and used 1.40 file set as 1.41 didn't work... The part am confused is where they tell us to put this

adb shell mkdir -p /sdcard/Android/data/com.{yourorgname}.{yourappname}/filed

So you mean, the name is any name we want, or facebook linked name... org name we create on oculus developer account
 what about the yourappname, what if I just wanted to record mrc content, not develop an app, how do you access the yourappname?, We put any name?

For adb push, we suppose push to our quest, first step and second step, 
according to instructions, we suppose keep same yourorgname.yourappname,
 then supposably suppose open app after finishing obs installation and then 
click connect, should load beatsaber...so unsure why put the above
com.beatgames.beatsaber

someone on YouTube who posted beatsaber with quest...
but unsure of authenticity or if he had private link to their account...or
developed his own way to connect...

[Lazius001]

Here's his link https://youtu.be/ZcWQaK01C-g

[Lazius001]

Or you mean for yournameorgname.yourappname, we put the name_of_any_developer.theirappname of app we want to connect too, so not necessarily our nameorgname.our_own_app name if we dont have an app?

Maybe that's where I got confused...

[rib]

hmm, curious, maybe BeatSaber does have mixed reality support (or they used something else like LIV if that maybe also supports Quest somehow?). I got the impression from the video linked above (from what the devs said) and from trying to connect to BeatSaber from OBS myself that it didn't yet ship with the necessary support.

For the naming; it's the name that corresponds to the specific application that you want to connect to (that must have this feature integrated first). It relates to the physical location where the application is installed on the device. For BeatSaber I found that it was installed under /sdcard/Android/data/com.beatgames.beatsaber. If you make your own Unity project then the default name is com.DefaultCompany.NewUnityProject. If you go to the Player Settings and look under the Android settings there is a Package Name, where it can be changed.

Whatever you do in Unity is only going to be related to recording your own developed application that you can side load onto the Quest. This new technology from Oculus for mixed reality with Quest requires all game developers to first integrate their MR technology into their game before it's possible to make recordings. Unfortunately it doesn't let you modify or record existing applications until their developers release an update with this extra functionality integrated. So for example the only way to record BeatSaber in mixed reality this way will be if they have released an update that enables this feature (and maybe they have already done that, but I didn't think they had yet).

Hope that makes sense - sorry if I have any wires crossed though 🙂