Showing results for 
Search instead for 
Did you mean: 

Research on body motion data storage technology

Level 2

========= Research subject =========

○ AI-based dance motion feedback using XR devices
  - Development of motion video recording technology using smartphone camera
  - Save the video itself in video format
  - Save the outline data of the motion using OpenCV(linked list format)
  - Using OpenCV, image frame-by-frame shading to store black and white shaded frame data(saved for use with Tensorflw's CNN model)
  - Saving the vertices of the image frame using OpenCV
  - Save the image file by dividing the video into frame units


========= Research conclusion =========

- In order to use the motion capture data of people in the video, various libraries of OpenCV were used to analyze and process the video.
- When data is imported as a data structure such as a point or line, the data capacity becomes smaller.
Although there are advantages, there are problems in using data
- The problem was that there were many difficulties in the evaluation method of dance only with points and lines.
In particular, there are many problems with the division of limbs or data length.
- As a result, after a lot of research, it was decided to save the image itself in frame units even if the amount of data is large.
- Successful research on image frame unit storage


========= Research subject =========

○ Deep learning of motion data using Google Tensorflow
  - Utilize motion analysis technology transferred from Jeonju National University (use motion analysis technology to classify motion scores, etc.)
  - Data analysis provides a library to OpenCV to study behavior analysis using OpenCV


<Research and Problems of OpenCV Behavior Analysis>
  ※ Create a pose by removing the background
  1. If the video has a solid background, it is estimated that the outline of a person will be found if the solid background is erased.
  2. Although it is a single color, many colors that are difficult to measure due to light reflection are generated.
  3. Analysis can only be performed in a specific space


  ※ Human body estimation through color
  1. Face detection in cascade face detector
  2. Detecting the skin color with the most color from the detected face
  3. Tracking the detected skin color (hands, neck, etc.)
  4. Using cascade (top, bottom) detector
  5. Extract the most color from the detected area
  6. Slowing down with multiple detectors
  7. It is difficult to detect if the color of the clothes is various
  8. Color can be lost to the background


  ※ Detection through Canny
  1. Attempt to find by outline with quick processing and removing color variability
  2. Too many outlines and the outlines are cut off by light bleed
  3. Difficulty in defining contours


  ※ OpticalFlow
  1. When the color and background of a person are similar, it is difficult to capture the details, so it is impossible to measure the details.
  2. The speed is slow, so additional video editing is difficult.


  ※ MeanShift, CamShift (high-speed tracking algorithm for objects of interest)
  1. It has fast processing speed and high tracking rate
  2. It is difficult to re-trace if the target to be tracked has to be selected, so if the target disappears from the video or the target to be tracked is obscured


  ※ Chromakey
  1. After saving the background without people, when a person enters, try to find another part and create an outline
  2. Noise caused by lighting
  3. When the camera (cell phone) shakes, the color coordinates change, making tracking difficult (it may shake during dance)


  ※ BackGroundSubtractorMOG2 (captures moving objects compared to previous data)
  1. By measuring the movement of a fixed camera, it is extracted by determining that a moving object is a human being.
  2. Difficult to measure due to external movement (other people or moving objects)


  ※ Cartoon Effect
  1. Divide the area by cartooning the image
  2. Unusable due to slow speed


  ※ water shed
  1. Divide the area through the water shed
  2. Segmentation but too many divisions in an image with many colors and slow speed

    - Research using Google Tensorflow's model instead of OpenCV
    - Data learning data learning with CNN model of Tensorflow learns image data that has been shaded to reduce the amount of learning
    - The CNN model has no problem in learning, so the resolution of the shadow data is low and it is difficult to distinguish the motion, so the learning result is very poor, so the study is stopped
    - Started a new study of Tensorflow's PoseNet model
    - PoseNet is a model specialized in human motion recognition and boasts a very fast speed compared to the CNN method in the case of existing human motions.
    - However, Posenet also has an analysis time of 2 fps per second based on the Galaxy Note 8, and it takes 12 seconds to analyze 1 second of video based on 24 frames.
    - Analysis time greatly exceeds the original reference time, but it is difficult to find any more optimized models, so we decided to use PoseNet and study optimization and refactoring.
    - Conduct code research and optimization to remove unnecessary parts and increase speed by analyzing PoseNet's learning data analysis and processing pipeline
    - Succeeded in PoseNet improvement research and succeeded in increasing the processing speed of the initial 2fps to 10fps


========= Research conclusion =========

- I studied the analysis library of OpenCV to analyze the image data, but No matching library found
- The CNN model of Google Tensorflow was also studied, but the study was terminated because it was difficult to distinguish the body due to learning the background, and there were serious problems with capacity and speed.
- Google PoseNet is slow at 2 fps, but has an advantage in terms of capacity.
※ Focusing on this early 2 fps increase the speed to 10 fps research success