For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/10/developing-an-embedded-vision-ai-powered-fitness-system-a-presentation-from-peloton-interactive/ Sanjay Nichani, Vice President for Artificial Intelligence and Computer Vision at Peloton Interactive, presents the “Developing an Embedded Vision AI-powered Fitness System” tutorial at the May 2023 Embedded Vision Summit. The Guide is Peloton’s first strength-training product that runs on a physical device and also the first that uses AI technology. It turns any TV into an interactive personal training studio. Members have access to Peloton’s instructors, who lead classes that use dumbbells and body weight. The Guide uses an innovative combination of hardware, software and user interface; it is one of only a handful of consumer products in this market that use real-time embedded computer vision, and Peloton’s members love it! In this talk, Nichani shares insights into how the computer vision technology behind the Guide was built. He focuses on data, models and processes. The AI and Computer Vision team at Peloton worked through numerous obstacles by using synthetic data, advanced algorithms and field-testing iteration to create an affordable AI-powered device that enables at-home fitness for everyone.

1. Developing an
Embedded Vision AI
Powered Fitness System
Sanjay Nichani
VP, Artificial Intelligence & Computer Vision
Peloton Interactive

2. The Peloton Guide
Transform any television in your home into
an immersive boutique fitness studio
powered by AI
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.

3. Computer Vision Features
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Activity Recognition and Movement Tracker
Tracks members and recognizes their
activity, as they follow along with
instructors to complete moves in the class.
Smart Framing and Self Mode
Guide digitally pans and zooms so you can
see yourself on screen and compare your
form to the instructor’s form.
3

4. Computer Vision Features
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Rep Tracker
Rep Tracking not only checks that user
is performing the correct exercise, but
also counts how many reps are done.
4
Time Tracked Moves
Allows user to follow exercise led by
instructor and receive real-time credit for
completing movement.

5. Other Machine Learning Features
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Body Activity
Body Activity shows the users various
muscles worked out and provides
personalized full body workout
suggestions.
5
Voice Control
Help members easily navigate content, find
classes, control weights and manage their
workouts with voice → fast and hands free.

6. Machine Learning Flywheel
● Person detect/tracking
● Pose estimation/match
● Activity recognition
● Rep counting
● Ensemble
● On-device
Modeling
● Open source
● Custom data
● Labeling/Tagging
● Synthetic data
•
Datasets
● Test datasets
● Attribute coverage
● Live lab testing
● Live field testing
● Production telemetry
Evaluate
● Data/Label ingestion
● Data transform / QA
● Compute optimization
● Tracking / Versioning
● Deployment monitoring
Infrastructure
ML iteration is key to success: helps to
find data gaps, data quality issues and
enables focused model enhancements
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved. 6

7. Datasets
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Video QA Label QA
Data
Sourcing
and
attribute
Tagging
Data
Labeling
and
attribute
Tagging
Reference
material creation
Synthetic datasets
Real-world datasets
7

8. CV Algorithms
8
Deep Match
Template matching using
metric learning
Deep Move
Action recognition with
temporal shift buffers
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.

9. Diversity Attributes for Metrics
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Geometric attributes
03
● Camera/Member relative placement
● Camera mounting Height
● Camera tilt
● Member orientation
● Member distance
Member attributes
02
● Gender, skin tone
● Body type
● Fitness level
● Disabilities
● Clothing
Environmental attributes
01
● Background: walls, flooring, furniture, windows
● Lighting, shadows, reflections
● Other people, animals in the field of view
● Occlusion
● Equipment (mat, dumbbells)
9

10. Fairness Metric
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Equality of opportunity: For a preferred label (specific exercise is being
performed) and a given attribute, a classifier predicts that preferred label equally
well for all values of that attribute. True Positive Rates are shown in table below.
Exercise Median Attribute: Body Type Fitzpatrick Skin Tone Gender
Underweight Average Overweight 1-2 3-4 5-6 Man Woman
Crossbody Curl 97% 100% 100% 97% 100% 98% 100% 100% 98%
Squat 99.7% 100% 100% 100% 100% 100% 100% 100% 100%
Dumbbell Swing 96% 100% 96% 98% 98% 100% 97% 97% 100%
10

11. Robustness Driven through UI and Nudging
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
Edge cases nudging:
• Brightness
• Occlusion
• Camera tilt, distance
• Multiple people
Orientation nudging occurs in all
time-based and rep tracking
workouts to minimize inaccuracies
due to occlusion.
11

12. Conclusion
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
• Rapid ML iteration helps find data gaps, data
quality issues and enable focused model
enhancements
• Diversity attributes are important to ensure
balanced training, but also key to slice and dice
performance
• ML and UI work together to ensure that
member has a robust experience without
introducing friction
• Both objective metrics focused on worst case
performance + subjective metrics
measurements are necessary to drive
improvements
12

13. Resources
© Peloton 2012-2023, Peloton Interactive Inc. All Rights Reserved.
The Peloton Guide
Peloton AI principles
Blazepose
Temporal shift module
Early stage recommender systems
Context aware recommender systems
User experience platform for connected fitness
13