Up and Coming

Working PS Move plugin for UE4; need help coreg with DK2

180 replies
100200 views

Update:

The plugin is not done but it's ready for public testing. I made a new thread here.

Hello all,

I recently made a PSMove plugin for UE4 that works in both Windows and OSX. It should work in Linux too with a simple compile of the psmoveapi binaries. The changes I had to make to the psmoveapi and PS3EYEDriver should also help anyone that wants to use the PSMove controllers in Windows or OSX (it's always worked in Linux) outside of UE4.

I need help co-registering the coordinates I get from the psmove (in cm from the camera) with the DK2's coordinate space.

For orientation, the roll and pitch are easy. The PSMove has a magnetometer so it's possible to get its true yaw but to make that useful in game I would also need access the DK2's magnetometer (I've read mixed reports that it even has one). Does anyone have any ideas here? Does it truly have a magnetometer? If so, does anyone know how to access it from within UE4?

If it does not then I'll need to somehow coregister the orientations by relying on the user to somehow point the controller in a specific orientation and pressing a button. I'm not too excited about that option.

For position, I would love it if I could access the DK2's camera's image feed. I could have the users run a quick calibration app that asked them to draw in the air then for each frame I would estimate the PSMove controller's position in 3D space relative to each camera, then create a transformation matrix from PS3Eye coordinates -> DK2 camera coordinates. This matrix could be stored in a data file somewhere (or registry entry) and as long as the cameras don't move relative to each other then it should stay the same. Any ideas on accessing the DK2's images, even outside of a game engine or the SDK?

If I can't access the DK2's camera feed then I'll probably need to touch the PSMove at specific locations on the surface of the DK2 and sample >=4 fiducial locations. I would need accurate dimensions of the DK2 body relative to whatever origin
it uses. Anyone have any experience with this?

Accuracy is important to me. We've all read that accuracy goes a long way to helping with immersion, so I hope that it's important to you too, but it's especially important to me because I'm using the data for research. So I can't really accept calibration methods that simply ask the user to put the controller near where they think it should be in the world and press a button.

Show previous replies

F

Fredz

Up and Coming

"bullale" wrote:
The new tracker algorithm returns values in cm, so do you even need to worry about scale?

It shouldn't be necessary but since it can be calculated cheaply and can be used to validate that everything is correct (scale 1) I'll probably implement it.

"bullale" wrote:
Did you ever find out how to get the HMD magnetometer reading from the SDK?

No but I've access to the rotation. I'm not sure the magnetometer is even used for that.

"bullale" wrote:
you thought about how to get the rotation (especially pitch) of the PSMove itself aligned to the same zero-pitch as the HMD?

Not yet, as I said previously I'll test if the coregistration is correct for positions that don't have rotation. I'll see afterward how I can use rotations as well.

"bullale" wrote:
Anyone on the sidelines that knows of a simpler way to get at the x,y,z values in world coordinates please let me know.

Looks good to me, the calculation isn't really time consuming.

B

bullaleAuthor

Up and Coming

Sorry I meant "Yaw" in the previous post.

Unlike pitch and roll, yaw is not affected by gravity and therefore can only be calculated by integration (subject to drift) of accel+gyro from some arbitrary starting point (press a button to reset orientation!) unless a magnetometer (or external camera) gives an absolute reference to the yaw and eliminates drift.

When a user is dropped into a game, typically their character body will be facing a certain direction dictated by the game but the HMD will be facing an arbitrary direction depending on the yaw of their chair.

There are a few ways to correct this.

Require the user to turn their head (swivel chair) so they are facing the correct direction. Not practical with wired HMDs.

Require the user to use arrow keys or mouse-look to yaw their character body to the correct orientation to compensate for the HMD yaw. This works when the character has no visible body.

Provide an HMD-yaw-reset function in the SDK (i.e., not handled by the game engine other than capturing the keypress and sending the reset command). From then on the HMD will return 'relative' yaw.

For the first option, nothing needs to be done to the PSMove. It'll just work as long as the HMD and the PSMove have the same yaw=0 direction (e.g. magnetic North).

For the second and third options, take a look at the attached diagram.
Yaw_PSMove_HMD.png

I'm pretending the game yaw=0 is +Z and that both the HMD and PSMove raw yaw=0 is magnetic North. In either case, I'm saying that clockwise is positive angles.

We want psi (green): the yaw of the PSMove in game relative to +Z. From the psmoveapi we can get phi (yellow): the PSMove's yaw relative to magnetic North. The difference is delta (blue): the yaw offset between magnetic North and +Z.

We can get delta from the HMD. From the HMD we should be able to get alpha (red): its raw world yaw. From the HMD and/or Engine we should be able to get theta (purple): the HMD yaw in game. The difference between those values is delta.

Note that this value will change every time the user reorients themselves with arrow keys, mouse l/r, or HMD-Orientation-Reset. So it needs to be calculated on every frame.

F

Fredz

Up and Coming

"bullale" wrote:
Unlike pitch and roll, yaw is not affected by gravity and therefore can only be calculated by integration (subject to drift) of accel+gyro from some arbitrary starting point (press a button to reset orientation!) unless a magnetometer (or external camera) gives an absolute reference to the yaw and eliminates drift.

Yes I know about that, IIRC in the DK2 the magnetometer isn't used and yaw drift is corrected by the camera. It doesn't seem to be possible to reliably correct yaw drift with magnetometers from what I've read. The DK1 and the Gear VR both suffer from it and I think it's going to be a problem for the PS Move as well, although to a lesser extent for hand tracking vs head tracking.

"bullale" wrote:
Provide an HMD-yaw-reset function in the SDK (i.e., not handled by the game engine other than capturing the keypress and sending the reset command). From then on the HMD will return 'relative' yaw.

I think that's the best approach, a recenter button is often used in DK2 demos and it's probably going to be a requirement when using the PS Move.

"bullale" wrote:
Note that this value will change every time the user reorients themselves with arrow keys, mouse l/r, or HMD-Orientation-Reset. So it needs to be calculated on every frame.

Ah yes, I didn't think about that. I had only envisaged demos without artificial locomotion for now, but that's clearly limiting. It's going to be a bit invasive, but we should probably try to think about an API for different engines to account for this.

B

bullaleAuthor

Up and Coming

"Fredz" wrote:

Yes I know about that, IIRC in the DK2 the magnetometer isn't used and yaw drift is corrected by the camera. It doesn't seem to be possible to reliably correct yaw drift with magnetometers from what I've read. The DK1 and the Gear VR both suffer from it and I think it's going to be a problem for the PS Move as well, although to a lesser extent for hand tracking vs head tracking.

When it is properly calibrated, the PSMove drift is really not bad. I guess HMDs are a little worse because it is more disruptive to the senses and because there is (probably) more electromagnetic interference.

"Fredz" wrote:
"bullale" wrote:
Note that this value will change every time the user reorients themselves with arrow keys, mouse l/r, or HMD-Orientation-Reset. So it needs to be calculated on every frame.

Ah yes, I didn't think about that. I had only envisaged demos without artificial locomotion for now, but that's clearly limiting. It's going to be a bit invasive, but we should probably try to think about an API for different engines to account for this.

I'm not positive that this number can even be calculated. We need to get the raw yaw and the 'corrected' yaw from the ovr_sdk on every frame.


// at some point in the past, ovrHmd_RecenterPose(ovrHmd hmd); was called
ovrTrackingState my_ts = ovrHmd_GetTrackingState(ovrHmd hmd, double absTime);
ovrQuatf game_orient = my_ts.HeadPose.ThePose.Orientation;
ovrSensorData raw_sens = my_ts.RawSensorData;
// ovrVector3f raw_sens.Accelerometer;
// ovrVector3f raw_sens.Gyro;
// ovrVector3f raw_sens.Magnetometer;

It's trivial to get the game_yaw from game_orient. I don't know how to get the raw_yaw from raw_sens. Do we really have to implement our own sensor fusion? I assume that, internally, the API is calculating the orientation using the raw data then adjusting it based on some value stored when recenterpose was called. It would be great if we could access the pre-adjusted raw orientation (and the stored orientation too). Maybe it's time to bring someone from Oculus into the conversation.

Probably the short-term solution will be to use a PSMove button to do double duty of recentering HMD and PSMove. This has to be done in the game engine. When the button is pressed, the yaw of the character body and the yaw of the PSMove are both saved to a variable. Then on each frame the game_yaw = raw_yaw - offset_yaw - body_yaw.

yaw yaw yaw.

edit: I posted the question separately.

F

Fredz

Up and Coming

"bullale" wrote:
When it is properly calibrated, the PSMove drift is really not bad. I guess HMDs are a little worse because it is more disruptive to the senses and because there is (probably) more electromagnetic interference.

Yes that makes sense. When I tried the PS Move just after my modification to the magnetometer calibration utility I found the drift to be quite minimal, but with the testing I'm doing at the moment I find it not that good. Maybe I need to recalibrate from time to time.

"Fredz" wrote:
I'm not positive that this number can even be calculated. We need to get the raw yaw and the 'corrected' yaw from the ovr_sdk on every frame.

I'm not sure I really understand what is really needed, I'll need to think about it a bit more or to be confronted to the problem in a real situation. Creating a function that must be called in real-time for each rotation initiated by a controller (mouse, PS Move buttons, recenter key, etc.) seems to be enough to me, but I may be wrong.

I'm not even sure that it should be required, since I can get the position of both the DK2 and the PS Move in camera space, so they'll always be correctly co-registered until the DK2 camera or the PS Eye is moved. Then a new co-registration would be required anyway.

For the rotation problem during the co-registration, I think it can be resolved by simply rotating the PS Move positions by the angle given by the DK2 tracker around the DK2 pivot point (which should be the DK2 position that has been read).

I didn't get a useful rotation from my calculations using the Horn method, the initial matrix is symmetric so there is no problem of row/colum major order. I'll try by simply calculating the average rotation for all the correlated points to see if I can at least obtain a useful result.

With the DK2 and PS Eye camera oriented in the same direction the corrected positions are quite close already anyway, but they diverge when they are not close from the center. I guess the biggest problem for now is the orb tracking code, so after getting the average rotation I'll try to implement your algorithm using the intrinsic parameters from the camera to get something more useful before trying to tackle again the calculation of the rotation with co-registration.

B

bullaleAuthor

Up and Coming

Edit: Deleted diagram and essentially incorrect information. See here for latest tracker algorithm.

B

bullaleAuthor

Up and Coming

The new tracking algorithm is now working in my new_tracker branch of the psmoveapi. For now the relevant functions are suffixed with _cbb (my initials). This is necessary because the old functions returned the center of the sphere x,y and radius in pixels whereas I do not even bother to calculate those anymore. I return only the 3D coordinates of the sphere in real-world centimetres. If I replace the old functions with my new functions then anything else expecting pixel results will break.

F

Fredz

Up and Coming

Great! :) I'll compile that and see what results I get in my calibration app.

EDIT:

I've adapted my code to use the new tracker, using psmove_tracker_update_cbb for updating and psmove_tracker_get_location to get the xyz coordinates, that I divide by 100 to get meters. I guess these are the only needed modifications since it shows correctly in my app.

Unfortunately it's quite jerky for now compared to the previous implementation so it's difficult to compare. I guess it may be related to a change in the exposure level or because the contour detection maybe isn't using subpixel estimation (I didn't look at the code yet).

There is also still a lot of distortion at the edges but I probably need to calibrate my PS Eye first or use the xml files from another calibrated camera.

EDIT 2:

I've tested by copying the distortion.xml and intrinsics.xml files from your previous post in my C:\Users\<Name>\AppData\Roaming\.psmoveapi folder but the distortion seems to be the same.

B

bullaleAuthor

Up and Coming

The original tracker had smoothing built-in to it. I've added that. It only has a history of one frame. The constants it uses may have to be tweaked.

For calibration, now that I've updated the tracker_calibration app to build again, you should be able to build it and calibrate your camera yourself. All you have to do is print the pattern.png

F

Fredz

Up and Coming

I've printed the calibration pattern and I get these values :

<?xml version="1.0"?>
<opencv_storage>
<intrinsics type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>f</dt>
  <data>
    5.43812927e+002 0. 2.94034058e+002 0. 5.43812927e+002
    2.25897202e+002 0. 0. 1.</data></intrinsics>
</opencv_storage>

<?xml version="1.0"?>
<opencv_storage>
<distortion type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>f</dt>
  <data>
    -1.33484483e-001 3.60221535e-001 6.41055347e-004 2.04809988e-003
    -4.67526495e-001</data></distortion>
</opencv_storage>

Show more replies

Get Help

Sign up

Login with SSO

Login to the community

Login with SSO