Pose Extraction for Real-Time Workout Assistant: Milestone 3

1. POSE
EXTRACTION
FOR REAL-TIME
WORKOUT
ASSISTANT
COURSE IT4983 IT
CAPSTONE
Royce Camp
Zachary Christmas
Jonathon Segars
Amanda Mead
Cameron Page
Milestone Report
#3
April 23, 2021

2. PROJECT
SUMMARY
The purpose of this project is to find, test, and
compare a software solution that deals with pose
extraction during real-time workouts. Our team
will choose a software solution, analyze its
effectiveness in extracting pose information from
images and videos, and then extract the
coordinates for key points of data from those
videos. We will then plot these coordinates using
a visualization platform to see how the
coordinates change during the video. We have
two external software's we will compare to our
chosen software solution and compare our
results.

3. MILESTONE 3
GOALS
 Expand on our testing, analysis, and key point
tracking for pose extraction.
 Continue to finalize and polish our PowerBI report
with visuals and more data.
 Compare our chosen software solution against
two others for performance in terms of running
speed and accuracy on different types of exercise
videos.

4. THE CHOSEN SOFTWARE SOLUTION: OPENPOSE
 Download OpenPose from: https://github.com/CMU-Perceptual-Computing-
Lab/openpose/releases

5. OPENPOSE - CONTINUED
 Provided the most accurate pose extraction
 Built in methods to extract the coordinates of key points
 Works on Windows and Linux
 Leading solution in the pose estimation open-source market

6. OPENPOSE
OVERLAY
 To run OpenPose, and
output a video with the
overlay: navigate to the
openpose file, and run
the command:
 binOpenPoseDemo.exe --
video [input video file path
with .mp4 or .avi video] --
net_resolution 320x320 --
write_json output_folder/ -
-write_video [output folder
name]/[output video title]

7. OPENPOSE POSE
EXTRACTION
WITHOUT OVERLAY
 Openpose also allows you to
extract just the skeleton of the
pose without overlaying it on
the video.
 binOpenPoseDemo.exe --video
[input video file path with .mp4
or .avi video] --net_resolution
320x320 --disable_blending --
write_json output_folder/ --
write_video [output folder
name]/[output video title]

8. OPENPOSE POINT
EXTRACTION
 To extract the keypoints
and export them as a
single file: navigate to
the openpose file, and
run the command:
 binOpenPoseDemo.exe --
video [input video file path
with .mp4 or .avi video] --
net_resolution 320x320 --
write_coco_json [output
folder name]/ [keypoint file
name .json]

9. CONVERT JSON
TO CSV
 Once the JSON has been
output from OpenPose,
we have data for all
available keypoints that
can be extracted.
 Here is a sample of 3
frame outputs from
OpenPose.
 As you can see, it is
difficult to identify what
keypoint is associated
with a given body part

10. CONVERT JSON
TO CSV -
CONTINUED
 To work around this issue,
a JavaScript function can
be written to extract and
label the keypoints that we
are concerned about

11. CONVERT JSON TO CSV - CONTINUED
 We took it a step further and placed that function in a public-facing web application where anyone
could extract data from OpenPose.
 Built using Create React App (https://github.com/facebook/create-react-app)
 Deployed using Git version control and Netlify: (https://www.netlify.com/)
 Source code can be found here (https://github.com/zachristmas/extract_openpose_json)
 Public facing web application can be found here: (https://nostalgic-ride-ca6d37.netlify.app/)
 Simply follow the instructions on the webpage to get the data for the keypoint you need.

12. IMPORTING CSV TO POWERBI
 Each CSV file is initially isolated to each key
point and video.
 We combined each CSV into one CSV file for a
whole video.
 Once I had a single file for a set of points for
each video, I used that data in the dashboard.
 Once the information was loaded in PowerBI I
was able to build out the X and Y vertical and
horizontal movement over each frame to
visual on its dashboard.
 Two pages were added for X, Y, Z mappings
from Fastpose.

13. POWERBI DASHBOARD

14. POWERBI DASHBOARD CONTINUED

15. FASTPOSE_DRNOOD
LE SOLUTION
 Lightweight 2D/3D
Realtime Pose Estimation
 Leverages AI-powered
deep-learning algorithms
to render 3D keypoints
using only a single RGB
camera
 3D Pose extraction can be
run on an external server
and fed to any web-capable
client

16. VIBE: VIDEO
INFERENCE FOR
HUMAN BODY
POSE AND SHAPE
ESTIMATION
 3D video pose and shape estimation software
 Supports multiple AI algorithms each for a different use case
(multi person vs single person)
 Can use pre-trained models or train models from scratch
 New feature allows pose extraction data to output FBX and
gITF files for use in 3D graphics applications like Blender and
Unity
Note: keypoints are output via
"tensors" to render a full human-
shape model, so extracting data
for specific body points was not
feasible.

17. EXTERNAL SOLUTIONS COMPARED TO OPENPOSE –
INSTALL AND EXECUTE
 Fastpose was easier to install and setup, as it was setup in a virtual environment.
 Openpose document was thorough but was not as easy to install.
 JSON to CSV conversion was similar, but we were able to extract the Z coordinate from Fastpose unlike
Openpose.
 Vibe was difficult to get running as it had a lot of dependencies.
 Vibe was able to provide a 3D pose extraction model but no points.
 Vibe had tensors but was not possible to get human readable body part location points.
 Fastpose seemed to render frames a lot quicker than Openpose; This seemed to sacrifice accuracy.
 Unable to get 3D (Z) point extraction from Openpose.

18. EXTERNAL SOLUTIONS COMPARED TO OPENPOSE CONT. -
PERFORMANCE AND ACCURACY
 Data points extracted from OpenPose were
more accurate and precise than Fastpose in a
frame-to-frame comparison.
 As you can see in the screenshot here on the
PowerBI report the Fastpose data points are
more jagged and inconsistent when compared
to OpenPose.
 The data points on the OpenPose were more
fluid.
 Fastpose was easier to execute than OpenPose
but you can see the performance and accuracy
took a hit.

19. PROBLEMS
ENCOUNTERED
 Access
 Many solutions require the Human 3.6m dataset
 Human 3.6m dataset access is limited
 Application process and access can take several months or longer
 Common errors encountered
 Competing drivers
 Poor documentation
 Abundance of software solutions
 Unable to extract datapoints
 VIBE Specific
 Pickled tensors format resulted in hundreds of points
 Challenging to focus in on a specific moment of motion
 Openpose Specific
 Required special cameras in order to function as 3D
 Capable of 3D, but only as a live application

20. PROJECT TIMELINE – GANNT CHART

21. QUESTIONS & ANSWERS