Pose Extraction for Real-Time Workout Assistant: Milestone 2

1. POSE
EXTRACTION
FOR REAL-TIME
WORKOUT
ASSISTANT
COURSE IT4983 IT
CAPSTONE
Royce Camp
Zachary Christmas
Jonathon Segars
Amanda Mead
Cameron Page
Milestone Report
#2
February 14, 2021

2. PROJECT
SUMMARY
The purpose of this project is to find, test, and
compare a software solution that deals with pose
extraction during real-time workouts. Our team
will choose a software solution, analyze its
effectiveness in extracting pose information from
images and videos, and then extract the
coordinates for key points of data from those
videos. We will then plot these coordinates using
a visualization platform to see how the
coordinates change during the course of the
video.

3. MILESTONE 2
GOALS
 Apply the best pose extraction software out of the
solutions we tested in Milestone 1
 Extract coordinates of key points of the pose
extracted from each frame of the video
 Visualize the sequence of the extracted
coordinates over time

4. THE CHOSEN SOFTWARE SOLUTION: OPENPOSE
 Download OpenPose from: https://github.com/CMU-Perceptual-Computing-
Lab/openpose/releases

5. OPENPOSE - CONTINUED
 Provided the most accurate pose extraction
 Built in methods to extract the coordinates of key points
 Works on Windows and Linux
 Leading solution in the pose estimation open-source market

6. OPENPOSE
OVERLAY
 To run OpenPose, and
output a video with the
overlay: navigate to the
openpose file, and run
the command:
 binOpenPoseDemo.exe --
video [input video file path
with .mp4 or .avi video] --
net_resolution 320x320 --
write_json output_folder/ -
-write_video [output folder
name]/[output video title]

7. OPENPOSE POINT
EXTRACTION
 To extract the keypoints
and export them as a
single file: navigate to
the openpose file, and
run the command:
 binOpenPoseDemo.exe --
video [input video file path
with .mp4 or .avi video] --
net_resolution 320x320 --
write_coco_json [output
folder name]/ [keypoint file
name .json]

8. CONVERT JSON
TO CSV
 Once the JSON has been
output from OpenPose,
we have data for all
available keypoints that
can be extracted.
 Here is a sample of 3
frame outputs from
OpenPose.
 As you can see, it is
difficult to identify what
keypoint is associated
with a given body part

9. CONVERT JSON
TO CSV -
CONTINUED
 To work around this issue,
a JavaScript function can
be written to extract and
label the keypoints that we
are concerned about

10. CONVERT JSON TO CSV - CONTINUED
 We took it a step further and placed that function in a public-facing web application where anyone
could extract data from OpenPose.
 Built using Create React App (https://github.com/facebook/create-react-app)
 Deployed using Git version control and Netlify: (https://www.netlify.com/)
 Source code can be found here (https://github.com/zachristmas/extract_openpose_json)
 Public facing web application can be found here: (https://nostalgic-ride-ca6d37.netlify.app/)
 Simply follow the instructions on the webpage to get the data for the keypoint you need.

11. IMPORTING CSV TO POWERBI
 Each CSV file is initially isolated to each key
point and video.
 We combined each CSV into one CSV file
for a whole video.
 Once I had a single file for a set of points
for each video, I used that data in the
dashboard.
 Once the information was loaded in
PowerBI I was able to build out the X and Y
vertical and horizontal movement over each
frame to visual on its dashboard.

12. POWERBI DASHBOARD

13. POWERBI DASHBOARD CONTINUED

14. PROBLEMS ENCOUNTERED
 Difficulty getting software packages installed in KSU provided lab computer
 Experienced problems getting competing software solutions installed/running to compare
 Encountered issues with computer video card needed to run differing pose extraction videos
 Adding more advance visual features to PowerBI.

15. MILESTONE 3 GOALS
C-Day Final report
Expand on the PowerBI tool (add
video and more visuals).
Field experiment on comparing
the chosen open-source package
with professional KinaTrax.
Other items to be determined by
project sponsors

16. PROJECT TIMELINE – GANNT CHART