Skip to main content
bullale
Up and Coming
March 15, 2015

Working PS Move plugin for UE4; need help coreg with DK2

Update:

The plugin is not done but it's ready for public testing. I made a new thread here.


Hello all,

I recently made a PSMove plugin for UE4 that works in both Windows and OSX. It should work in Linux too with a simple compile of the psmoveapi binaries. The changes I had to make to the psmoveapi and PS3EYEDriver should also help anyone that wants to use the PSMove controllers in Windows or OSX (it's always worked in Linux) outside of UE4.

I need help co-registering the coordinates I get from the psmove (in cm from the camera) with the DK2's coordinate space.

For orientation, the roll and pitch are easy. The PSMove has a magnetometer so it's possible to get its true yaw but to make that useful in game I would also need access the DK2's magnetometer (I've read mixed reports that it even has one). Does anyone have any ideas here? Does it truly have a magnetometer? If so, does anyone know how to access it from within UE4?

If it does not then I'll need to somehow coregister the orientations by relying on the user to somehow point the controller in a specific orientation and pressing a button. I'm not too excited about that option.

For position, I would love it if I could access the DK2's camera's image feed. I could have the users run a quick calibration app that asked them to draw in the air then for each frame I would estimate the PSMove controller's position in 3D space relative to each camera, then create a transformation matrix from PS3Eye coordinates -> DK2 camera coordinates. This matrix could be stored in a data file somewhere (or registry entry) and as long as the cameras don't move relative to each other then it should stay the same. Any ideas on accessing the DK2's images, even outside of a game engine or the SDK?

If I can't access the DK2's camera feed then I'll probably need to touch the PSMove at specific locations on the surface of the DK2 and sample >=4 fiducial locations. I would need accurate dimensions of the DK2 body relative to whatever origin
it uses. Anyone have any experience with this?

Accuracy is important to me. We've all read that accuracy goes a long way to helping with immersion, so I hope that it's important to you too, but it's especially important to me because I'm using the data for research. So I can't really accept calibration methods that simply ask the user to put the controller near where they think it should be in the world and press a button.

    180 replies

    Show previous replies
    bullaleAuthor
    Up and Coming
    June 29, 2015
    "Fredz" wrote:
    Registration of the positional tracking without orientation is unfortunately useless for the use of the PS Move as a hand-tracked controller.


    Not for me. I'm just moving a sphere through space, orientation doesn't matter. But orientation data would be richer so it's something I hope to fix eventually. I think a good short-term solution will be to use Madgwick but set the yaw=0 point with a button press. This can already be done, but there's still substantial yaw drift, so it needs to be reset often.
    bullaleAuthor
    Up and Coming
    July 7, 2015
    My psmove_dk2_coreg utility now works. It writes a csv file to the home directory. (Currently no error checking).

    That csv file is then loaded by psmove_fusion to make a transformation matrix. The transform is applied on every frame when using psmove_fusion_get_transformed_position.

    That only brings you to the DK2 camera reference frame. From there you have to figure out how to go from DK2 camera reference frame to game world. The basic idea is that you get the DK2 camera in DK2-HMD coordinates with ovrHmd_GetTrackingState (note that you need to use CameraPose, not the LeveledCameraPose), figure out the necessary transforms from DK2-HMD reference frame to game world reference frame, then commute those transforms and send the result to the psmoveapi with psmove_fusion_update_transform.

    This is turning out to be really difficult in UE4.

    One reason is that Oculus plugin for UE4 doesn't actually call ovrHmd_RecenterPose, it simply updates an internal representation of the current pose and orientation whenever the UE4 function ResetOrientationAndPosition is called, then subtracts that on every frame. That means I need to pull out the internal representation too, but I can't access it directly. I haven't figured out how to include the Oculus Plugin in my module so I can access its internal variables.

    It looks like this should be considerably easier in Unity, especially now with the Utilities extension.
    Fredz
    Up and Coming
    July 8, 2015
    Nice that you got this integrated into the PS Move API, too bad for the UE4 support. Do you intend to implement the plugin for Unity or do you want me to do it ?
    bullaleAuthor
    Up and Coming
    July 8, 2015
    I'm not very familiar with Unity so I'd rather you do it if possible.

    One nice thing about UE4 is that I can hack the source code, which I did to expose the member variables I need. I'm now working through the rest. I think I'm close... I hope.
    bullaleAuthor
    Up and Coming
    July 8, 2015
    It works! It's not perfect but it's pretty good. I doubt my UE4 plugin C++ code will be of much use to you, but here's the relevant section.

    I mostly followed what Oculus did for rendering the DK2 camera in the game world, but applied that to the psmove position. It seems like it could be made easier and I'm not sure all of the offsets are necessary.

    One thing that's not obvious from that chunk of code is that I feed the PSMoveComponent object the location and orientation of the player camera on every frame. This is necessary if the player moves off the initial position of 0,0,0 and away from the initial orientation of yaw=0 (e.g., using mouse and keyboard), otherwise the controller will stay in the same place in the game world and will not follow the player.
    bullaleAuthor
    Up and Coming
    July 9, 2015
    I've updated it since that last post. The code is now a lot cleaner, better commented, and it works better. Here.

    I still haven't fully figured out the orientation.
    Fredz
    Up and Coming
    July 9, 2015
    For the orientation I've found two algorithms that should be able to return an absolute rotation in an earth reference frame using only the accelerometer and the magnetometer :

    - Accelerometer and Magnetometer Based Gyroscope Emulation on Smart Sensor for a Virtual Reality Application (Delporte et al. 2013)
    - A Simplified Quaternion-Based Algorithm for Orientation Estimation From Earth Gravity and Magnetic Field Measurements (Yun, Bachmann, McGhee 2008)

    I've not implemented them yet.

    Once the orientation is initialized with one of them, I think the Madgwick algorithm should work without singularities problems using the gyroscope + accelerometer + magnetometer. I think the registration of the orientation would then be quite straightforward.
    bullaleAuthor
    Up and Coming
    July 9, 2015
    Can you remind me what you're trying to solve with that? I'm not getting singularities. In fact, the rotation works just fine for me. The only problem I have is yaw drift. If I put the controller in the yaw=0 direction and press the yaw_reset button, it works well for 2-3 seconds until the yaw drift creeps in.
    Fredz
    Up and Coming
    July 9, 2015
    Yaw drift is not too bad for me, but not all initial physical orientations of the PS Move are in the same calculated orientations, which is a problem for the registration of the PS Move and DK2 orientations.

    When I place the PS Move with the orb at the top and the buttons toward me, the orientation is correct each time. But when I place the PS Move horizontally with the buttons on top it's visually not represented like that, it's almost vertical with a ~20° angle wrt gravity.

    With the accelerometer and magnetometer it should be possible to get the correct orientation each time, so I think there is a problem with singularities in the Madgwick algorithm. I guess he went for a faster algorithm to calculate the acc + mag orientation because he probably intended his algorithm to work correctly with a gyroscope, for which singularities don't seem to be a problem (or maybe that's the cause of the yaw drift).

    The two algorithms I linked to are specifically designed to avoid singularities and are able to provide an orientation in an earth reference frame (NED convention).

    With this it should be possible to find the transformation between the PS Move orientation and the DK2 orientation calculated with sensors. The DK2 orientation in the DK2 camera reference frame is known as well, so it should be possible to find the transformation between it and the DK2 orientation calculated with sensors. Applying both these transformations to the PS Move should give its rotation in the DK2 camera reference frame.

    Calculation of these transforms should probably be done with the Horn algorithm ("absolute orientation") I've implemented previously for more robustness or a similar one (because this one seems to suffer from some singularities in specific cases as well).
    bullaleAuthor
    Up and Coming
    July 18, 2015
    Here's a quick video demonstrating how well (or not well) it works for me. You can see the yaw drift.

    The coregistration was done before recording this video. I added some smoothing to my tracking algorithm. It definitely reduces the jitter, but it also adds latency :( . I might disable XY smoothing and only keep the z-smoothing. Having added latency in only one dimension might be less noticeable.

    I forgot to push all the changes before I left work. I'll do that on Monday. Nevertheless, you'd still need to modify the UE4 source to use it.




    Edit: Thanks for the youtube tag tip Cybereality
    Show more replies
    Terms & ConditionsAccessibility statement