Up and Coming

Working PS Move plugin for UE4; need help coreg with DK2

180 replies
100200 views

Update:

The plugin is not done but it's ready for public testing. I made a new thread here.

Hello all,

I recently made a PSMove plugin for UE4 that works in both Windows and OSX. It should work in Linux too with a simple compile of the psmoveapi binaries. The changes I had to make to the psmoveapi and PS3EYEDriver should also help anyone that wants to use the PSMove controllers in Windows or OSX (it's always worked in Linux) outside of UE4.

I need help co-registering the coordinates I get from the psmove (in cm from the camera) with the DK2's coordinate space.

For orientation, the roll and pitch are easy. The PSMove has a magnetometer so it's possible to get its true yaw but to make that useful in game I would also need access the DK2's magnetometer (I've read mixed reports that it even has one). Does anyone have any ideas here? Does it truly have a magnetometer? If so, does anyone know how to access it from within UE4?

If it does not then I'll need to somehow coregister the orientations by relying on the user to somehow point the controller in a specific orientation and pressing a button. I'm not too excited about that option.

For position, I would love it if I could access the DK2's camera's image feed. I could have the users run a quick calibration app that asked them to draw in the air then for each frame I would estimate the PSMove controller's position in 3D space relative to each camera, then create a transformation matrix from PS3Eye coordinates -> DK2 camera coordinates. This matrix could be stored in a data file somewhere (or registry entry) and as long as the cameras don't move relative to each other then it should stay the same. Any ideas on accessing the DK2's images, even outside of a game engine or the SDK?

If I can't access the DK2's camera feed then I'll probably need to touch the PSMove at specific locations on the surface of the DK2 and sample >=4 fiducial locations. I would need accurate dimensions of the DK2 body relative to whatever origin
it uses. Anyone have any experience with this?

Accuracy is important to me. We've all read that accuracy goes a long way to helping with immersion, so I hope that it's important to you too, but it's especially important to me because I'm using the data for research. So I can't really accept calibration methods that simply ask the user to put the controller near where they think it should be in the world and press a button.

Show previous replies

B

bullaleAuthor

Up and Coming

"Fredz" wrote:

Do you know where the image can be found ? It was mentioned that it was available in the PS Move API but I didn't find it.

I found something here. I remember reading one somewhere else that is meant to be printed on A4. Because the linked website is in the US I imagine it is meant to be printed on Letter (also what we use in Canada).

There's a png here which is maybe the same as the official OpenCV one.

F

Fredz

Up and Coming

I've read a bit more about the co-registration of two sets of 3D points, and it seems to be a classical problem in applied mathematics (often used in satellite attitude determination) known as the Wahba's problem. It can be solved with the Triad method or the Kabsch algorithm, the latter one giving the optimal solution which can be calculated using Singular value decomposition (SVD).

I've found a math library (Math.NET Numerics) that I was able to import in Unity and which does implement SVD, so I think I'll have a try at finding the transform matrix between the two set of points with the Kabsch algorithm.

The outline of the algorithm is described here :
- Kabsch algorithm - Computation of the optimal rotation matrix
- Finding optimal rotation and translation between corresponding 3D points
- Finding a Rotation Transformation from two Coordinate Frames in 3-Space

The steps don't seem as complicated as I first though, it goes like this :
- find the centroids (average point) of both sets of coordinates
- bring both sets of coordinates to the origin
- find the optimal rotation (matrix R) using SVD
- find the translation t = -R * centroidA + centroidB

For the two sets of points, I'll define a procedure for the user where he needs to move the DK2+PS Move in the FOV of the camera while orienting the DK2 perpendicular to the camera. Probably not the most optimal but it should help to validate the general algorithm and could still bring a good approximation for the co-registration.

F

Fredz

Up and Coming

I've implemented the Kabsch algorithm but it doesn't work yet. The final quaternion I obtain from the computed rotation matrix is incorrect (as told by Unity), I used the simple formula described here but I need to use the more complicated algorithm that is described afterwards. I'm not even sure the matrix I obtained is a real rotation matrix and that all the steps I've implemented are correct, so I'll try to convert it in the correct way to a quaternion and if that fails I'll try another implementation of the Kabsch algorithm (Horn method) which directly returns a quaternion and seems a bit simpler and more robust.

Working with the Math.NET Numerics library has been a real pain, the documentation is really sparse and the API has changed a lot over the years so it's really difficult to understand how to implement things with it. I thought libraries were supposed to simplify the life of a programmer, but with this one it's unfortunately not the case. In the end I may have to implement everything myself, that way I would be sure that I'll end up with correct calculations and have a deeper understanding about what happens.

Really sad to not have been able to get a solution during this 3-days week-end, I hope I'll be able to obtain some results this week.

B

bullaleAuthor

Up and Coming

This post is mostly for AlphaWolf:

I modified the tools in the `Binaries\Win64` folder of the plugin. I replaced the camera tester with `ps3eye_sdl.exe` which is a relatively simpler test of the camera. I removed psmovepair.exe and am now relying only on psmove-pair-win.exe. I also updated the Wiki to reflect the new instructions.

B

bullaleAuthor

Up and Coming

"Fredz" wrote:

Do you know where the image can be found ? It was mentioned that it was available in the PS Move API but I didn't find it.

The calibration app is in the psmoveapi examples folder. The chessboard png is in the contrib folder.

I haven't tested yet.

B

bullaleAuthor

Up and Coming

I have to run to catch a bus in just a second...
but I updated the psmoveapi so it can build the calibration app if PSMOVE_BUILD_EXAMPLES and PSMOVE_BUILD_TRACKER are set. You'll have to rebuild OpenCV with some additional settings.

I also attached the .exe here (needs psmoveapi_tracker.dll, I think).

I'm not sure it worked. I'll poke at it more from home.

tracker_camera_calibration.zip

F

Fredz

Up and Coming

@bullale: Great, I'll try that as soon as I'm finished with co-registration.

B

bullaleAuthor

Up and Coming

The tracker calibration app works in Windows and Mac now. Along the way I also modified the PS3EYEDriver so it is now possible to set the green balance.

Here's what I got for distortion.xml

<?xml version="1.0"?>
<opencv_storage>
<distortion type_id="opencv-matrix">
  <rows>5</rows>
  <cols>1</cols>
  <dt>f</dt>
  <data>
    -1.02539599e-01 1.54094994e-01 5.24583645e-03 2.65011028e-03
    -7.39907250e-02</data></distortion>
</opencv_storage>

and intrinsics.xml

<?xml version="1.0"?>
<opencv_storage>
<intrinsics type_id="opencv-matrix">
  <rows>3</rows>
  <cols>3</cols>
  <dt>f</dt>
  <data>
    5.54511963e+02 0. 3.36385651e+02 0. 5.54511963e+02 2.33075073e+02 0.
    0. 1.</data></intrinsics>
</opencv_storage>

F

Fredz

Up and Coming

I've implemented the registration for the rotation with the Horn method based on quaternions and now I'm able to get a correct quaternion for the orientation, at last. :) In the end it's way simpler than the SVD method and I've read everywhere that it's more robust, precise and numerically stable so I'll stick with that.

For anyone interested, the outline of the algorithm is :
- calculate the centroids for the arrays of recorded positions for the DK2 and the PS Move
- recenter the recorded positions around their respective centroid so we can find the rotation between the two arrays of positions independently of the translation or scale
- compute the matrices of sums of products for the two arrays of recorded positions
- compute the matrix representation of the quaternion that needs to be found (which minimizes the distance between arrays of positions) based on the previous calculations
- find the maximum positive eigenvalue of this 4x4 matrix
- find the eigenvector which corresponds to this eigenvalue
- copy this vector into a quaternion, which is the optimal rotation between the two arrays of recorded positions

I've not implemented the position and scale registration yet but I've found several papers explaining how it works (this one for example) and it seems quite straightforward compared to the computation of the rotation.

The orientation returns a correct quaternion but it's not correctly oriented yet. In my app I rotate the PS Move model with this calculated quaternion in the scene, but when I move the DK2 in a direction the PS Move moves in another direction. It moves at a similar pace and on an orthogonal axis so it's reassuring that the problem is probably only related to the row-major/column-major order for my matrices. I guess that can be hopefully easily solved with some trial and error.

Nice that you got the tracker calibration app working bullale, it should be very helpful for the implementation of the new tracking algorithm.

B

bullaleAuthor

Up and Coming

Edit: Deleted diagram and essentially incorrect information. See here for latest tracker algorithm.

Show more replies

Get Help

Sign up

Login with SSO

Login to the community

Login with SSO