Up and Coming

Working PS Move plugin for UE4; need help coreg with DK2

180 replies
100201 views

Update:

The plugin is not done but it's ready for public testing. I made a new thread here.

Hello all,

I recently made a PSMove plugin for UE4 that works in both Windows and OSX. It should work in Linux too with a simple compile of the psmoveapi binaries. The changes I had to make to the psmoveapi and PS3EYEDriver should also help anyone that wants to use the PSMove controllers in Windows or OSX (it's always worked in Linux) outside of UE4.

I need help co-registering the coordinates I get from the psmove (in cm from the camera) with the DK2's coordinate space.

For orientation, the roll and pitch are easy. The PSMove has a magnetometer so it's possible to get its true yaw but to make that useful in game I would also need access the DK2's magnetometer (I've read mixed reports that it even has one). Does anyone have any ideas here? Does it truly have a magnetometer? If so, does anyone know how to access it from within UE4?

If it does not then I'll need to somehow coregister the orientations by relying on the user to somehow point the controller in a specific orientation and pressing a button. I'm not too excited about that option.

For position, I would love it if I could access the DK2's camera's image feed. I could have the users run a quick calibration app that asked them to draw in the air then for each frame I would estimate the PSMove controller's position in 3D space relative to each camera, then create a transformation matrix from PS3Eye coordinates -> DK2 camera coordinates. This matrix could be stored in a data file somewhere (or registry entry) and as long as the cameras don't move relative to each other then it should stay the same. Any ideas on accessing the DK2's images, even outside of a game engine or the SDK?

If I can't access the DK2's camera feed then I'll probably need to touch the PSMove at specific locations on the surface of the DK2 and sample >=4 fiducial locations. I would need accurate dimensions of the DK2 body relative to whatever origin
it uses. Anyone have any experience with this?

Accuracy is important to me. We've all read that accuracy goes a long way to helping with immersion, so I hope that it's important to you too, but it's especially important to me because I'm using the data for research. So I can't really accept calibration methods that simply ask the user to put the controller near where they think it should be in the world and press a button.

Show previous replies

konstantinloze

Up and Coming

Awesome work! Any chance in helping the FreePIE community enable PS Move support?

bullaleAuthor

Up and Coming

"AlphaWolF" wrote:

And i noticed i didn't have the ps3eye_sdl.exe mentioned in the bottom of the page so i re-downloaded the latest psmove-ue4 and saw it was included with some other new files and the reg entry file had been removed .
Everything went fine and launching the ps3eye_sdl.exe I can actually see myself and get a picture :) even if its a slight bit more pixlelated than the CL driver version.
But it seems im having the same kind of issue again.

Launching the. tracker_camera_calibration.exe
= Warning: no lens calibration files found.

Launching the. test_tracker.exe
= ### Found 1 controllers.
Trying to init PSMoveTracker...Warning: No lens calibration files found.

I hope this is not a dead end again :(

I'm really glad to hear you got it working.

The ps3eye_sdl.exe has the camera properties set to 320x240 @ 187 Hz. It's supposed to look pixelated. All other programs use 640x480 @ 60 Hz. If you'd like to know why ps3eye_sdl uses different settings then you'd have to ask the person that made that modification to the ps3eye_sdl code in the up-upstream PS3EYEDriver.

The registry file is removed because I modified the code so it is no longer needed.

As for the lens calibration file not being found, that's just a warning. It doesn't stop anything from working. All it means is that you haven't run the tracker_camera_calibration application. You don't have to if you don't want to. If you do want to then you need to print out pattern.png and ... well, really you should follow directions from elsewhere on the web on how to do camera calibration with OpenCV. I've only done it a couple times and I think I'm not very good at it. After the camera is calibrated then every frame from the camera will be undistorted. This will give better tracker accuracy. But again, it's totally not necessary.

Fredz,
I'm sorry that I don't seem to be getting e-mails anymore whenever a new post goes up. I haven't read anything in a little while.

One of the reasons for posting that data is that it is straight from the respective APIs, without any axis inversions from any plugins. The only thing is that I set the tracker mirror mode to True, which causes each image to be cvFlipped around the y-axis, basically negating the x direction. When this is done, it seems that the pamoveapi_tracker and the DK2 use the same coordinate system (right handed).

Unity and UE4 both use left-handed coordinate systems. This means that the DK2 plugin must handle flipping one of the axes. And I flipped an axis in the PSMove plugin for UE4. My light reading of the DK2 plugin also make me think that it's doing some other less intuitive things. I think it might be a good idea to first get the coregistration working outside any of the game engines then worry about the plugin-specific transformations.

I'm super tired and falling asleep. I'll give your latest posts a deeper read and comment on the 3DOF vs 6DOF in the coming days.

AlphaWolF

Participant

I'm really glad to hear you got it working.

As for the lens calibration file not being found, that's just a warning. It doesn't stop anything from working. All it means is that you haven't run the tracker_camera_calibration application. You don't have to if you don't want to. If you do want to then you need to print out pattern.png and ... well, really you should follow directions from elsewhere on the web on how to do camera calibration with OpenCV. I've only done it a couple times and I think I'm not very good at it. After the camera is calibrated then every frame from the camera will be undistorted. This will give better tracker accuracy. But again, it's totally not necessary.

I was under the impression that the test_tracker.exe & tracker_camera_calibration.exe were actually supposed to do something ? or display some kind of tracker utility that i can see the wand being tracked by the PS EYE.
but All i see i the black cmd prompt box with the lens calibration files missing message & thats it!.
nothing happens beyond that message. just a warning message in a cmd box.

http://alphawolf-uk.deviantart.com/ https://twitter.com/AlphaWolF_UK http://steamcommunity.com/id/j_alphawolf_76/ https://www.youtube.com/user/AlphaWoIFuk/videos http://www.nexusmods.com/skyrim/mods/9198/ ?

bullaleAuthor

Up and Coming

"AlphaWolF" wrote:

but All i see i the black cmd prompt box with the lens calibration files missing message & thats it!.
nothing happens beyond that message. just a warning message in a cmd box.

tracker_camera_calibration.exe should display a Window representing the camera view, and when you make the chessboard pattern visible to the camera, the window will show some coloured lines and give you the option of pressing space to save the data. Do that 10 times then you're camera is calibrated. Is there any possibility that the window is being opened but is not showing up on your main screen?

Just to prove to yourself that the warning message is in fact a warning and is not responsible for any trouble you're having, please download the attache .zip file and extract the contents (two .xml files) into your

C:\Users\<username>\AppData\Roaming\.psmoveapi\

folder. The warning message should no longer show up.

Because it's been a while, can you check that the magnetometer_calibration.exe and test_calibration.exe work for you?

uname_appdata_roaming_psmoveapi.zip

bullaleAuthor

Up and Coming

Fredz,

I'm not sure where you're at with this at the moment.

I just noticed a small problem with your image, I think. It seems like the X coordinate values are reversed (shouldn't it be -0.7 for the PSMove and -0.5 for the DK2?).

The student I have working on this drew out the AX = YB equation for me with a diagram. He told me that the PSMove orientation will not matter if it's placed on right right of the multiplication (B). I didn't get this part but OK. This is good because we don't have any way to get its orientation in the PSEye coordinate frame. But he was dismayed because he said he needed the DK2 orientation in the DK2 camera coordinate frame, which we didn't have.

However, unlike the PSMoveAPI, the DK2 SDK returns the DK2 positional coordinates in a reference frame that has the X-Z plane normal to gravity. In other words, its position and tilt (pitch + roll) are all in the same coordinate frame. And Yaw is arbitrary. If you call ovrHmd_RecenterPose, it zeroes the position and it zeroes the yaw for both orientation and position. Now, facing whichever way you were facing when you zeroed the Yaw, translating your head left->right is +x, down->up is +y, and back->front is +z. This is good because it means that the position and orientation are in the same coordinate frame.

But this new reference frame is arbitrary (at least the origin and yaw=0 direction). Anytime you call ovrHmd_RecenterPose you'll be in a new reference frame, and your old solution to the coregistration will no longer work. Fortunately, we can get the camera's physical location and yaw in the new reference frame from ovrTrackingState.LeveledCameraPose. I guess it will be necessary to take the DK2 pose data in its arbitrary reference frame and transform it into a camera-centric reference frame.

Note, you could also use ovrTrackingState.CameraPose which gives the tilt too, but you would have to ignore the tilt when transforming the the DK2 data back into camera coordinates. The tilt might be useful if someone at some point designs a 3d-printable a mount for both the DK2 camera and PSEye camera.

bullaleAuthor

Up and Coming

Here are two datasets. The first one has a DK2 leveled camera pose of (px, py, pz, ox, oy, oz, ow):

17.657907, 1.003707, -67.863470, 0.000000, -0.116859, 0.000000, 0.993149

.

The second one has a DK2 leveled camera pose of:

79.909188, -25.432739, -106.208837, 0.000000, -0.315269, 0.000000, 0.949002

.

The first dataset was recorded around one physical location relative to the pair of cameras. The second dataset was recorded after I pushed my chair back, rotated a bit, and stood up. In both cases, I called ovrHmd_RecenterPose to get the camera pose before sampling the data. You can see that the second camera pose has a different yaw and the position is pretty different in all axes even though I only translated in 2.

I think the procedure should be to first solve the coregistration using output1. This will require first transforming the DK2 poses into the camera-centric reference frame.

Then, for output2, the obtained transformation matrices from output2 that get the PSMove position in the DK2-camera-centric reference frame will have to be modified, using the new leveled camera pose, to yield the PSMove position in the DK2-centric (not camera-centric) reference frame. In other words, every time the user calls ovrHmd_RecenterPose, the calculated transformations will have to be modified using the new leveled camera pose.

And just to clarify what I said above about ignoring the PSMove orientation. I think we can ignore the PSMove orientation data when calibrating (i.e., we can set the output1 quaternions to 0, 0, 0, 1) but we still expect the transformation from PSMove space to DK2-leveled-camera space to have rotational components, not just translation, because the camera will not be perfectly level and it will have non-zero yaw relative to the DK2 camera.

output1.txt

output2.txt

Fredz

Up and Coming

"bullale" wrote:
I'm not sure where you're at with this at the moment.

Not far, I think that the problem is not resolvable, but I can be wrong.

"bullale" wrote:
I just noticed a small problem with your image, I think. It seems like the X coordinate values are reversed (shouldn't it be -0.7 for the PSMove and -0.5 for the DK2?).

That's what is shown in the schema, maybe a cache problem, I remember I corrected that some days ago.

"bullale" wrote:
He told me that the PSMove orientation will not matter if it's placed on right right of the multiplication (B). I didn't get this part but OK.

I guess he's getting the inverse of the rotation matrix this way, which he can revert back to the original form by simply inverting it again.

"bullale" wrote:
But he was dismayed because he said he needed the DK2 orientation in the DK2 camera coordinate frame, which we didn't have.

I thought that the DK2 orientation was in the DK2 camera coordinate frame already, at least it's what it looks like when I show the DK2 in 3D. I guess I called a recenter pose at one time in my tests or maybe it's done automatically at each restart.

"bullale" wrote:
Note, you could also use ovrTrackingState.CameraPose which gives the tilt too

I'm using capiHmd.GetTrackingState().HeadPose.ThePose.ToPose() which seems to give the DK2 pose I was looking for.

Does your student think the problem is solvable ? Because I've been trying to understand it and I'm not sure it's possible.

Consider :
- Ra : DK2 rotation
- A : DK2 position
- B : PS Move position in PS Eye reference frame
- B' : PS Move position in DK2 camera reference frame
- Rx : rotation from PS Eye to DK2 camera
- Tx : translation from PS Eye to DK2 camera
- Ty : translation from PS Move to DK2 in DK2 camera reference frame

We have :
A = Inverse(Ra).B' + Ty
B' = Rx.B + Tx

Which gives :
A = Inverse(Ra).Rx.B + Inverse(Ra).Tx + Ty

This can be solved with the Horn or equivalent algorithm (A = R.B + t) by substituting :
R1 = Inverse(Ra).Rx
T1 = Inverse(Ra).Tx + Ty

So we have R1 and T1, we can calculate Rx since Ra is known, but we can't solve both Tx and Ty knowing only Ra and T1. We can calculate the global transform between A and B, but we can't discern the transform between B and B'.

Note that the equation I've written in my schema is wrong.

I guess there might be a way since AD7GD said this in the reddit thread :
"It turns out you can tell those two things apart because only one pair of transformations will work in every orientation. I tried to write out an intuitive explanation of why this is true, but it really needs diagrams."

But he didn't explain why nor how. I'll ask him if your student doesn't have the answer.

What also leads me to believe that there is not necessarily a solution for this problem is that I've not found algorithms for this particular case in the literature, only algorithms for 6dof data, whereas we only have 3dof data for the PS Move.

Maybe the QR15 algorithm from Ernst 2012 that is supposed to work with partial data (incomplete or no rotation) is a solution, but I didn't understand it.

When I asked AD7GD some weeks ago how he would tackle the problem, he talked about Bundle adjustment, not about closed-form solutions or classical linear/nonlinear least squares method, so I guess there is not necessarily a know solution to this problem if one even exists.

Fredz

Up and Coming

"bullale" wrote:
And just to clarify what I said above about ignoring the PSMove orientation. I think we can ignore the PSMove orientation data when calibrating (i.e., we can set the output1 quaternions to 0, 0, 0, 1)

I ignored the PS Move orientation as well, it's quite unreliable and it's not in a meaningful coordinate frame ("world" instead of PS Eye reference frame).

"bullale" wrote:
but we still expect the transformation from PSMove space to DK2-leveled-camera space to have rotational components, not just translation, because the camera will not be perfectly level and it will have non-zero yaw relative to the DK2 camera.

Yes I accounted for that as well (Rx and Tx in my equations).

bullaleAuthor

Up and Coming

"Fredz" wrote:

I thought that the DK2 orientation was in the DK2 camera coordinate frame already, at least it's what it looks like when I show the DK2 in 3D. I guess I called a recenter pose at one time in my tests or maybe it's done automatically at each restart.

No, the orientation and position are in an arbitrary coordinate frame that gets changed everytime ovrHmd_RecenterPose is called: the position origin is zeroed to the HMD position and the Yaw is zeroed to the direction the HMD is facing. Pitch and Roll are not changed because those are physical measurements relative to gravity. For the purpose of this conversation, we can call this HMD-centric, as long as we remember that the reference frame does not follow the HMD around after the call to ovrHmd_RecenterPose.

I'm pretty sure ovrHmd_RecenterPose is called automatically by UE4 and I guess it is also called automatically by Unity. I just tested calling it manually in my C code, then restarting the oculus service via the config utility, and it seems the values from before I restarted the service were preserved.

Fredz

Up and Coming

When I read the Ernst 2012 paper I found a notation I didn't understand that basically transformed the AX = YB problem into a AX = XB problem for which there are known solutions. Unfortunately this equation also requires 6dof data like the AX = YB problem.

But reading the Shiu 1989 paper it seems having a known rotation for B is not needed. When the AX=XB equation is separated into rotational and positional components it gives : RARX=RXRB

The paper demonstrates that RA must be equal to RB, so I don't need to know the orientation of the PS Move. And that seems to make sense, since if the PS Move is attached to the DK2 and the DK2 is rotated, the PS Move should be rotated with the exact same angle.

The translational component of the AX = XB equation doesn't even depend on RB either (R:rotation, P:position) :
RAPX+PA = RXPB+PX

I've summed up the equation in this reddit thread I posted some days ago (without answers unfortunately, but my question could have been more clear) : http://www.reddit.com/r/math/comments/3 ... er_with_a/

So I'll look for a MATLAB implementation for the AX = XB problem, convert it in C#, test it with synthetic data and if it works test it with real data.

Show more replies

Get Help

Sign up

Login with SSO

Login to the community

Login with SSO