Working PS Move plugin for UE4; need help coreg with DK2

Hi Fredz,

Do you have something I can try today? It doesn't have to be polished. I just want to make the changes to the psmoveapi to read the XML file and apply the transformation before I go away for a week. I'm leaving tomorrow.

TIA

Sorry I didn't see your post, I've just updated my Github repo with the Ernst 2012 calibration implementation and some tries with the calibration of the rotation using the Madgwick C# sensor fusion implementation, in case you've Internet access.

As you know I've been playing around with your utility a bit. I wonder if you can help clear up a couple questions.

First:
You use the Unity-specific ThePose.ToPose(). This automatically does a righthanded-to-lefthanded conversion of the pose (v.z = -v.z; q.x = -q.x; q.y = -q.y;). It appears UniMove does the same thing for position only, but negates the x. What has to be done to get the LHS-to-LHS transform generated by your utility to work at the API-level on RHS position data?

Second:
The HMD pose you get from the API is in its own reference frame that is reset whenever someone calls ovrHmd_RecenterPose. This might be done by the player, by the game, etc. I think it would be better, instead, to use the poses in the camera reference frame to obtain the transformation from PSMove camera-frame to DK2 camera-frame, because the cameras should not move relative to each other very often. Then the psmoveapi, when transformation is desired, will return the PSMove position in DK2 camera-frame. Finally, it will be up to the game or plugin to convert that position to DK2-frame.

How do we convert from DK2 frame to DK2 camera frame and vice versa? ovrTrackingState.LeveledCameraPose gives us the current camera position in the current DK2 frame. So what's the linear algebra to invert that so the camera pose has x,y,z & yaw = 0 and the DK2 pose is in the DK2 camera frame? What's the linear algebra to convert the PSMove position in DK2 camera frame to DK2 frame?

"bullale" wrote:
You use the Unity-specific ThePose.ToPose(). This automatically does a righthanded-to-lefthanded conversion of the pose (v.z = -v.z; q.x = -q.x; q.y = -q.y;). It appears UniMove does the same thing for position only, but negates the x. What has to be done to get the LHS-to-LHS transform generated by your utility to work at the API-level on RHS position data?

When I started using UniMove it didn't support positional tracking so I implemented that myself, that's what I released 10 months ago. The code you're asking for seems to be this one in MoveManager.cs :

#if PSMOVE_TRACKING_THP
				psmove_fusion_get_position(fusion, handle, ref px, ref py, ref pz);
				position.x = -px * 0.05f; position.y = py * 0.05f; position.z = pz * 0.05f;
#elif PSMOVE_TRACKING_CBB
				psmove_tracker_get_location(tracker, handle, ref px, ref py, ref pz);
				position.x = -px / 100.0f; position.y = py / 100.0f; position.z = -pz / 100.0f;
#endif

"bullale" wrote:
The HMD pose you get from the API is in its own reference frame that is reset whenever someone calls ovrHmd_RecenterPose. This might be done by the player, by the game, etc. I think it would be better, instead, to use the poses in the camera reference frame to obtain the transformation from PSMove camera-frame to DK2 camera-frame, because the cameras should not move relative to each other very often. Then the psmoveapi, when transformation is desired, will return the PSMove position in DK2 camera-frame. Finally, it will be up to the game or plugin to convert that position to DK2-frame.

My goal was indeed to get the HMD pose in the DK2 camera referential and when I was looking for such a function this one seemed to be the only one that fitted the requirement. If it's not the case with a recenter I'll try to find another function.

"bullale" wrote:
How do we convert from DK2 frame to DK2 camera frame and vice versa? ovrTrackingState.LeveledCameraPose gives us the current camera position in the current DK2 frame. So what's the linear algebra to invert that so the camera pose has x,y,z & yaw = 0 and the DK2 pose is in the DK2 camera frame?

From the Oculus Rift Developer Guide :
- CameraPose : the pose of the camera relative to the tracking origin.
- LeveledCameraPose : the pose of the camera relative to the tracking origin but with roll and pitch zeroed
out. You can use this as a reference point to render real-world objects in the correct place.

Both seem to use the same tracking origin but in the first case the orientation is unaltered, I guess that's why I've chosen this one.

"bullale" wrote:
What's the linear algebra to convert the PSMove position in DK2 camera frame to DK2 frame?

I'm not sure I understand your question, isn't it described in the Ernst 2012 paper I've been talking about in this thread ?

"bullale" wrote:
It looks pretty good to me. It would of course be better if the distance after coregistration was a constant (i.e., histogram had all counts in only one bin) but this will do for now. I'm guessing most of the deviation is due to the psmove tracker noise.

Yes I guess so, and also from the lower precision of the tracking compared to the DK2 and the fact that the time of arrival of raw data is not taken into account and that it's not fed at the frequency of the PS Eye. Also there may be a delay between the reading of the DK2 and PS Move positions the way I implemented it in Unity. I should probably read everything in the same function instead of using two game objects.

Btw, did you already implement a reliable sensor fusion that gives an orientation in an absolute earth reference frame ? I need this to calibrate the rotation between the DK2 and PS Move.

Sorry I haven't had time to work on orientation. I'm not sure when I will. It's useful to my experiment but not critical.

As for the other questions, sorry that I was unclear. Let's skip the coordinate system (RHS vs LHS) for now and first consider doing the co-registration in C (not Unity or UE4).

When collecting data for calculating the transform, we sample PSMove position in the PSEye reference frame, and we sample DK2 HMD position in the arbitrary DK2 reference frame.

The PSEye reference frame, when facing the PSEye, has +x to the right, +y is up'ish, and +z is along the primary axis of the camera. The x-z plane is not exactly perpendicular to gravity.

The arbitrary HMD reference frame always has the x-z plane perpendicular to gravity. By default, the HMD reference frame has x=0 in the middle of the lateral extent of the camera's frustum, y=0 is at the same physical height of the camera (ignoring its pitch and roll), z=0 is 1 m away from the camera at x,y=0, and yaw=0 is when the HMD is facing the camera while at the origin. However, this reference frame changes whenever ovrHmd_RecenterPose is called.

Because the DK2 reference frame changes arbitrarily and outside of our control, it is not useful to coregister the PSMove to whatever the current arbitrary DK2 reference frame is. Instead, what needs to be done is to coregister the PSMove to the DK2 camera reference frame, because the DK2 camera should not move often.

So, instead, on each sample we:
-get the PSMove position as above
-get the DK2 position and orientation in its arbitrary reference frame, then convert that to the DK2 camera reference frame.

One question I have is how to convert DK2 poses from arbitrary reference frame to DK2 camera reference frame. I guess this is as simple as getting the LeveledCameraPose (obtained in arbitrary reference frame), making a 4x4 matrix out of that, inverting that matrix, then applying the resulting transformation matrix to the DK2 poses.

The transformation learned this way will then allow us to get the PSMove coordinates in the DK2 camera reference frame. This is not useful in the game because the DK2 camera reference frame is not used in game. We therefore need to convert PSMove coordinates to game coordinates in 3 steps: 1) from PSMove to DK2 camera, 2) from DK2 camera to arbitrary HMD, 3) from arbitrary HMD to game. Step 1 is what we get from the coregistration, step 3 is game/engine specific but is trivial. How do we do step 2? I know we will have to get the LeveledCameraPose as above, but then what?

I was able to answer my own question by playing with the data in Matlab.

For the coregistration
1) Get the camera leveled pose and convert to a 4x4 matrix.
2) Invert the camera_4x4
3) For each sample of the DK2, inv_camera_4x4 * DK2_4x4
4) Solve as previously to get coreg_xform

Then, 'in game', at startup and whenever ovrHmd_RecenterPose is called, get the new camera leveled pose and convert to current_camera_4x4 (not inv).
On each tick, psm_xyz_in_hmd = current_camera_4x4 * coreg_xform * psm_xyz_in_pseye

That's solved. Now time to talk about coordinate systems (next post)

Let's say we do the coregistration in C, and we have a transform matrix saved to an XML file. Then, the game is running, we request the psmove position, what do we want it to return?

1) The psmove position in pseye space (RHS)?
2) The psmove position in dk2camera space (RHS)?
3) The psmove position in dk2hmd space (RHS)?
4) The psmove position in dk2hmd_game space (LHS)?

To go from 1->2, we need the transformation matrix loaded from a file. I think the best way to do this is to have the psmoveapi_tracker load it. Let's say we store it in a variable called transform1.

To go from 2->3, we need to know the dk2 camera leveled pose. This cannot be obtained directly by the psmoveapi. There are two options. The first is that the game can get the dk2 camera leveled pose and pass that to the psmoveapi, which will commute it with transform1 to get total_transform. The second option is that the plugin (UniMove or my UE4 plugin) can do this transform. Do you have any preference here?

To go from 3->4, we need to convert the psmove position similarly to how the engine converts the DK2 position. This is currently, and should continue to be, handled by the plugin.

Finally, the player/game_camera offset has to be added to this position. That's normally handled by the game designer. Any desire to do this in the plugin, and how would that work?

One more thing that adds to the confusion: The DK2 coordinates are in m, the psmoveapi_tracker coordinates are in cm.

If I ignore this when doing the coregistration then the transformed psmove coordinates (after coregistration transformation) will be in m, and the engine plugins may attempt to further scale them.

If I change the camera + dk2 coordinates into cm during coregistration, then the transformed psmove coordinates will also be in cm, and the plugins don't have to care about this, but they will have to care about this when sending the API the camera pose for the secondary transform. Maybe that's OK.

"bullale" wrote:
There are two options. The first is that the game can get the dk2 camera leveled pose and pass that to the psmoveapi, which will commute it with transform1 to get total_transform. The second option is that the plugin (UniMove or my UE4 plugin) can do this transform. Do you have any preference here?

No preference there. But if the coregistration is implemented in the PS Move API it could make sense to also add a function that can return the position of the PS Move in whatever referential needed by the application. That would mean a bit less work for each specific plugin (not limited to UE4 and Unity), and would probably be easier for maintainability.

"bullale" wrote:
Finally, the player/game_camera offset has to be added to this position. That's normally handled by the game designer. Any desire to do this in the plugin, and how would that work?

No idea yet, I didn't think about it that far. I thought this could be done automatically without any developer's intervention.

"bullale" wrote:
One more thing that adds to the confusion: The DK2 coordinates are in m, the psmoveapi_tracker coordinates are in cm.

I'm afraid I don't understand the problem. The code I posted in my previous answer specifically converts from cm to m when reading the PS Move position. Is it not enough ?

"bullale" wrote:
Sorry I haven't had time to work on orientation. I'm not sure when I will. It's useful to my experiment but not critical.

Too bad. :/ I'm still struggling with this.

From what I've read in several papers it should be possible to get an absolute orientation using only the accelerometer and the magnetometer, but unfortunately the Madgwick algorithm doesn't support that correctly. When I set the gyroscope values to zero and iterate several thousand times I don't get the correct virtual orientation depending on the real orientation of the PS Move (180/90° inversions). I'm still looking for an algorithm that can give the correct quaternion in any case (the link I posted on Github didn't work out), which I'll use for the initialization of the Madgwick algorithm.

Registration of the positional tracking without orientation is unfortunately useless for the use of the PS Move as a hand-tracked controller.