Skip to main content
bullale
Up and Coming
March 15, 2015

Working PS Move plugin for UE4; need help coreg with DK2

Update:

The plugin is not done but it's ready for public testing. I made a new thread here.


Hello all,

I recently made a PSMove plugin for UE4 that works in both Windows and OSX. It should work in Linux too with a simple compile of the psmoveapi binaries. The changes I had to make to the psmoveapi and PS3EYEDriver should also help anyone that wants to use the PSMove controllers in Windows or OSX (it's always worked in Linux) outside of UE4.

I need help co-registering the coordinates I get from the psmove (in cm from the camera) with the DK2's coordinate space.

For orientation, the roll and pitch are easy. The PSMove has a magnetometer so it's possible to get its true yaw but to make that useful in game I would also need access the DK2's magnetometer (I've read mixed reports that it even has one). Does anyone have any ideas here? Does it truly have a magnetometer? If so, does anyone know how to access it from within UE4?

If it does not then I'll need to somehow coregister the orientations by relying on the user to somehow point the controller in a specific orientation and pressing a button. I'm not too excited about that option.

For position, I would love it if I could access the DK2's camera's image feed. I could have the users run a quick calibration app that asked them to draw in the air then for each frame I would estimate the PSMove controller's position in 3D space relative to each camera, then create a transformation matrix from PS3Eye coordinates -> DK2 camera coordinates. This matrix could be stored in a data file somewhere (or registry entry) and as long as the cameras don't move relative to each other then it should stay the same. Any ideas on accessing the DK2's images, even outside of a game engine or the SDK?

If I can't access the DK2's camera feed then I'll probably need to touch the PSMove at specific locations on the surface of the DK2 and sample >=4 fiducial locations. I would need accurate dimensions of the DK2 body relative to whatever origin
it uses. Anyone have any experience with this?

Accuracy is important to me. We've all read that accuracy goes a long way to helping with immersion, so I hope that it's important to you too, but it's especially important to me because I'm using the data for research. So I can't really accept calibration methods that simply ask the user to put the controller near where they think it should be in the world and press a button.

    180 replies

    Show previous replies
    bullaleAuthor
    Up and Coming
    April 12, 2015
    I just updated the tracker. There was a small bug.
    Fredz
    Up and Coming
    April 12, 2015
    It seems better with this modification. Your tracking implementation is more accurate than the previous one, there is less distortion when the orb is near the edges of the camera and in my app this results in less circular motion on the z axis.

    But it's still very jerky for me, a lot more than the previous implementation which was mostly stable. I tried using a different color for the orb (cyan) but it didn't change anything. It mostly happens when the orb is not in the central FOV of the camera, but it also happens (although less often) when it's close to the center. When there is this problem the position alternates between two set of values rapidly.

    The algorithm also seems to fail when the orb is not a complete sphere, which happens when the handle is occluding a small part of the sphere, but I guess it's only an edge case that has not been accounted for yet.

    Did you experience jerkiness yourself?

    I'll try to play a bit with the algorithm to try to find where it does come from.
    bullaleAuthor
    Up and Coming
    April 12, 2015
    "Fredz" wrote:
    But it's still very jerky for me, a lot more than the previous implementation which was mostly stable.

    Does "previous implementation" mean my previous version or the original (i.e. thp's)?

    "Fredz" wrote:
    When there is this problem the position alternates between two set of values rapidly.
    Did you experience jerkiness yourself?


    I haven't yet set up a way to evaluate the data easily. I've only been using the test_tracker program.

    I'm not terribly surprised it's alternating. I had that behaviour in some of my earlier attempts that mostly had to do with me incorrectly re-centering the ROI. I thought I fixed it, but maybe I only made the problem smaller. Try looking at that first.

    It might also be worthwhile to change the smoothing constant. This is currently the same value that thp's tracker uses for pixel coordinates, but I'm applying it to world-coordinates in cm. I'm sure the value can be tweaked.

    But I think a better way to compare this algorithm to the older one would be to disable smoothing in both. Smoothing should be the last thing to work on, and personally I think it should be up to each developer to decide how much to smooth as smoothing adds latency.
    Smoothing only happens if tracker->tracker_adaptive_z is true, so change the #define value it inherits from to 0.
    Fredz
    Up and Coming
    April 12, 2015
    "bullale" wrote:
    Does "previous implementation" mean my previous version or the original (i.e. thp's)?

    Yes thp's original, sorry for being unclear.

    "bullale" wrote:
    I'm not terribly surprised it's alternating. I had that behaviour in some of my earlier attempts that mostly had to do with me incorrectly re-centering the ROI. I thought I fixed it, but maybe I only made the problem smaller. Try looking at that first.

    I've debugged the code a bit and it seems that the problem happens because there is no sub-pixel edge detection as I feared. I see the calculated coordinates for the bounding box oscillating between two successive values, which produces some millimeters of difference, hence the jerkiness.

    It seems there is no algorithm in OpenCV for sub-pixel edge detection, so some specific code should be implemented. I've some ideas for naive implementations that I'll try, hoping that more complex (and costly) techniques won't be needed.
    bullaleAuthor
    Up and Coming
    April 13, 2015
    If we get the contour, uncomment these two lines, then use OpenCV fitellipse, we can get a CvBox2D object that contains the angle of the ellipse as well as its center and the length of its major and minor axes, but I don't think these results are subpixel.

    It might be necessary to fit a full ellipse to the data to get all its coefficients (a,b,c,d,e,f), but I think this is an expensive process. (maybe cheaper way). We could then use some calculus to find the bounding rectangle (minima and maxima) with sub-pixel accuracy. Of course, once we have the ellipse coefficients, there is probably some cheaper trigonometry to use to get the sphere coordinates.

    I wonder if there are some assumptions we can make to speed up the ellipse fitting. For example, does the major axis of the ellipse always point toward the origin (image midpoint)? Does the ellipse eccentricity scale (non-)linearly with the distance-to-origin?
    Fredz
    Up and Coming
    April 13, 2015
    "bullale" wrote:
    If we get the contour, uncomment these two lines, then use OpenCV fitellipse, we can get a CvBox2D object that contains the angle of the ellipse as well as its center and the length of its major and minor axes, but I don't think these results are subpixel.

    It could be useful if it can handle the case of partially occluded ellipses, a subpixel technique could then be applied on the edges along the major and minor axes.

    I've started to implement subpixel fitting with a naive technique (average of intensities in a 10x1 window, calculation of the intersection between the curve of intensities and the line of average intensity), I'll see if it gives more stable results.

    "bullale" wrote:
    I wonder if there are some assumptions we can make to speed up the ellipse fitting. For example, does the major axis of the ellipse always point toward the origin (image midpoint)? Does the ellipse eccentricity scale (non-)linearly with the distance-to-origin?

    Yes it seems to be useful information, in a way similar to an overconstrained system, but I'm not sure it's going to be simple to incorporate that in an algorithm.
    Fredz
    Up and Coming
    April 13, 2015
    I've implemented sub-pixel fitting for the left side of the bounding box and it's already more stable this way. Not visible with the test_tracker utility because I'm displaying a sphere at discrete positions, but visible in my PS Move Setup utility.

    Before doing it with the top, right and bottom side I'll try the OpenCV fitellipse function you mentioned because my code only tests the center of each side of the bounding box which is not necessarily where the point of contact is located.

    Hopefully it'll also help with the distortion at the edges which is still a problem in my test app. That is, if it's not a problem of camera calibration.
    bullaleAuthor
    Up and Coming
    April 15, 2015
    I'm using a new tracking algorithm. See here. I'll keep my figures on that site so I don't have to keep uploading them here.

    It works very well for me. With my controller stationary, I get 0.4 mm jitter in x and y, and ~1.5 mm jitter in z.

    I can see the bounding box alternating between pixels but that has no little/no effect on the tracker.
    Fredz
    Up and Coming
    April 15, 2015
    Nice. I'll try to backport my sub-pixel fitting (which I've extended to the 4 sides) to see if some precision can be gained and to compare the results in my test app. The DK2 has 0.05 mm precision so the closer to this the better I think.

    Now it would be interesting to see if the longer axis of the ellipse passes on the calibrated center of the camera as well.
    bullaleAuthor
    Up and Coming
    April 15, 2015
    "Fredz" wrote:
    Nice. I'll try to backport my sub-pixel fitting (which I've extended to the 4 sides) to see if some precision can be gained and to compare the results in my test app. The DK2 has 0.05 mm precision so the closer to this the better I think.

    I don't think we'll ever get to 0.05 mm precision with this technology. According to this article, in a conversation with a Sony employee

    Delving into a bit of tech talk, the designer said the camera has a field of about 75 degrees, with XY tracking down to millimeters and Z-axis tracking to a few centimeters.

    I think we're already doing better than that.

    I'm not sure if sub-pixel fitting will gain you anything over the fitEllipse. fitEllipse returns the ellipse center, rectangle, and angle in float, not pixel ints. Here is its algorithm.

    "Fredz" wrote:

    Now it would be interesting to see if the longer axis of the ellipse passes on the calibrated center of the camera as well.

    I printed the angle fitEllipse returns and the angle calculated by taking atan2f(x,y) and they were off by 1-3 degrees when the controller was in the upper right quadrant. When the controller was in another quadrant they were very different, but not quite 180 degrees different.

    Since the last message I updated the annotation function to draw the fit ellipse. You can see that it matches the blob very well and it doesn't jitter around, even though the ROI jitters around.

    When I occlude part of the sphere (either with my hand or with the controller itself by rotating it away from the camera), the ellipse x-y center is approximately the same but the fit ellipse is much smaller, and therefore the algorithm thinks it is much further away. Also, when this happens, the ellipse is definitely not pointing at the center.

    I'm interested to see how it works with your coregistration program. Do you think you could give me some metrics on how well it works vs thp's original algorithm? Maybe the RMS of the error? If we can get some hard numbers then maybe it'll motivate the upstream maintainer to finally merge my changes.

    Do you think I can have access to your coregistration program by this weekend, even if it's incomplete? I'm hoping to use the PSMove in an experiment next week.
    Show more replies
    Terms & ConditionsAccessibility statement