Wearables and hearables that measure biometric signals like heart rate are different from other devices, because they have to interact with the human body and every human body is different. This makes testing and validation of the devices an important part of the product development process. Valencell operates one of the most experienced testing labs for biometric wearables and hearables, testing hundreds of devices over thousands of hours of testing every year.

1. Best Practices in Testing Biometric Wearables
© 2017 Valencell, Inc
Chris Eschbach, PhD, Director Biometrics Laboratory

2. © 2017 Valencell, Inc
Webinar outline
Biometrics Laboratory overview
• Biometrics team
• Lab workflow and charge
Data collection process
• Areas for consideration
Data analysis and reporting

3. © 2017 Valencell, Inc
Webinar outline
Biometrics Laboratory overview
• Biometrics team
• Lab workflow and charge
Data collection process
• Areas for consideration
Data analysis and reporting

4. © 2017 Valencell, Inc
The Valencell Biometrics Lab is a cutting edge biometric
science lab designed to provide comprehensive analysis of
the latest in biometric sensor and wearable technology.
• Team of exercise scientists lead by PhD Exercise
Physiologist Dr. Chris Eschbach
• Pool of nearly 100 volunteers testing prototypes and new
technology every week
• Validation datasets on participants of numerous body
types, fitness levels, gender, & skin tones
• Testing protocols that match the use cases - resting,
lifestyle activities, mild exercise, aggressive exercise,
interval training, etc.
• Testing facilities for running, biking, swimming, gym
activities, and lifestyle activities
Valencell Biometrics Lab Overview
Biometrics Lab testing each year:
• Conducts over 24,000 different device
tests
• Analyses over 12 million biometric data
points
• Measures over 1,200 hours of testing &
validation sessions

5. © 2017 Valencell, Inc
Valencell Biometrics Lab Overview

6. © 2017 Valencell, Inc
Biometrics laboratory operations & staff
Operations
• Exercise sessions (R & D +
validation): 40 per week
• Exercise sessions (validation
commercial/training programs): 20
per week
• Other trials (longitudinal and rest):
30 per week
Staff
• Director
• Operations lead
• Data lead
• Reports lead
• Trials specialists

7. © 2017 Valencell, Inc
Biometrics Laboratory Workflow
• Scheduling
• Prioritization
• Experimental design
• Device/Experiment intake
• Reporting

8. © 2017 Valencell, Inc
Webinar outline
Biometrics Laboratory overview
• Biometrics team
• Lab workflow and charge
Data collection process
• Areas for consideration
Data analysis and reporting

9. © 2017 Valencell, Inc
Data collection process
Internal or external trials and projects
• Participants
• Data collection
• Areas for consideration
Data analysis and reporting

10. © 2017 Valencell, Inc
Participants
• Requirements: age, health, and availability
• Participant characteristics
• Health history
• Laboratory introduction
• Anatomical measures
• Pictures
• Goal for all DUTs: 20 participants with 8-min dynamic treadmill test

11. © 2017 Valencell, Inc
Participants- anatomical measures

12. © 2017 Valencell, Inc
Participants- Fitzpatrick Skin Type

13. © 2017 Valencell, Inc
Participants - comfort and security
Comfort
Rating
Comfort Description
1 • Painful and extremely uncomfortable.
• Worse than any other earbuds you have
used.
• Would not wear longer than 15 minutes.
2 • More uncomfortable than preferred.
• Below average compared to other
earbuds.
• Would not wear longer than 1 hour.
3 • Neither comfortable nor uncomfortable.
• Average compared to other earbuds.
• Would wear up to 2 hours.
4 • Comfortable.
• Above average compared to other
earbuds.
• Would wear up to 3 hours.
5 • Extremely comfortable.
• Better than any other earbuds you have
used.
• Would wear for 4 hours or longer.
Security
Rating
Security Description
1 • Won’t stay in ear.
• Worse than any other earbuds you have
used.
2 • Will stay in place while still, but slides out of
place with movement.
• Below average compared to other earbuds.
3 • Feels secure at rest and moderately insecure
while running but doesn’t fall out.
• Average compared to other earbuds.
4 • Feels secure at rest and stable while running
but falls out with a slight tug on the cord.
• Above average compared to other earbuds.
5 • Extremely secure at rest and while running.
• Better than all other earbuds.

14. © 2017 Valencell, Inc
Data collection process
Internal or external trials and projects
• Participants
• Data collection
• Areas for consideration
Data analysis and reporting

15. © 2017 Valencell, Inc
Data collection process
• Metrics collection
• Raw data
• Heart rate, RRi, steps, blood pressure, etc.
• Calories, VO2, Speed, Distance, etc.
• Baseline devices
• Chest strap HRM, ECG, Video, Manual and
automatic BP
• Indirect calorimetry, measured courses, and
calibrated treadmills

16. © 2017 Valencell, Inc
Data collection process - laboratory tools
• Wide range of participants
• Experienced staff
• Databases and data experts to overcome data collection
and UX challenges
• Data collectors (Raw, multi BLE, single BLE): commercial
applications and custom
• Exercise equipment
• Treadmills, bikes, strength, lifestyle, etc
• Reference devices (baseline and procedures for them)
• Collection devices: phones, cameras, programs, computers

17. © 2017 Valencell, Inc
Data collection process - laboratory tools
Reference devices (baseline and procedures for them)

18. © 2017 Valencell, Inc
Data collection process
• Protocol matches the intended use case(s)
• Participant places devices (followed by instruction as needed)
• Metrics collection- Heart rate
• DUT - Ideally raw data collected for analysis
• If not, then BLE HR
• Lastly some proprietary HR output
• Post processing to provide heart rate at 1 second intervals

19. © 2017 Valencell, Inc
Data collection process - protocols
Data is collected for the following 8-minute protocol:

20. © 2017 Valencell, Inc
Data collection process - protocols

21. © 2017 Valencell, Inc
Data collection process - protocols
• Various field trials beyond the foundational protocol (8-min dynamic treadmill)
• Sunlight testing
• Lifestyle (typical daily movements)
• Resting heart rate
• Dynamic bike indoor (10-minutes)
• Running intervals (15-minutes)
• Free run/bike (>20 minutes)
• Rowing ergometer (13-minutes)
• Strength circuit (typical strength exercises)
• High intensity interval strength
• Swimming

22. © 2017 Valencell, Inc
Data collection process - CTA protocols
• Consumer Technology Association
• Co-chair, Health Fitness and Wellness Technology Standards Committee
• Physical Activity Monitoring for Fitness Wearables: Step Counting
• Definitions and Characteristics for Wearable Sleep Monitors
• Methodology of Measurement for Features in Sleep Tracking Consumer
Technology Devices and Applications
• Physical Activity Monitoring for Heart Rate
• Consumer Stress Monitoring Technologies
https://cta.tech/Research-Standards/Standards-Listing/Health-Fitness-Technology-Standards.aspx

23. © 2017 Valencell, Inc
Data collection process – areas of consideration
Source: “Optical heart-rate measurement’s top 5 challenges” Dr. Steven LeBoeuf; EDN Magazine; 8-25-15
Optical Noise Skin Tone Blood Perfusion
Sensor Location Crossover Problem

24. © 2017 Valencell, Inc
Webinar outline
Biometrics laboratory overview
• Biometrics team
• Lab workflow and charge
Data collection process
• Areas for consideration
Data analysis and reporting

25. © 2017 Valencell, Inc
Data analysis and reporting
• Both subjective and quantified scoring of data
• Provides confidence with all data
• Easily understood during design cycles
• Erratic data: physiologically unlikely in a health
individual.
• Highly variable heart rate “jitter”
• Reported heart rate outside of expected ranges
(for example, reported at 0 or greater than 200)

26. © 2017 Valencell, Inc
Data analysis and reporting
• Excellent
• Signal was strong and tracking good the entire session
• Good
• Signal was strong and tracking was good but occasional short segments
of mis-tracking occurred
• A moderately scored test would likely not be noticed by a consumer
• Lost tracking
• Segments of moderate to long duration where biometric was not correct

27. © 2017 Valencell, Inc
Data analysis and reporting
Subjective results:
• Reference device
• DUT
• Excellent: >90% within +/-5%
• Good: >80% within +/-5%
• Poor: <80% within +/-5% and/or mis-tracking outside of +/- 10% for
greater than 40 seconds

28. © 2017 Valencell, Inc
Data analysis and reporting
• Statistical analysis
• mean ± SD (when appropriate)
• bias as percent error
• mean absolute percent error (MAPE)
• frequency distribution within ±5% windows
• Utilize thresholds: 85% of data will be within ±5%
• Bland-Altman and correlation as needed
• accuracy related to heart rate intensity (≤ moderate and ≥ vigorous)

29. © 2017 Valencell, Inc
Data analysis and reporting – areas for consideration
• Baseline errors (remove but report errors)
• Time for HR lock
• Data alignment
• Latency

30. © 2017 Valencell, Inc
Validation Report - representative data setsInitials
(1st 2 last
2) Unit ID Date and time Picture Core Gel Size
Ear Tip
Size
Needs
Warm-up Good Mod Fail
GeSc XXX5.58.3.M#8 7/10/14 10:50 M 1
CiSh
DoCa
HuRo XXX5.58.3.M#8 7/10/14 14:44 M 1
FaSa
JuSa XXX5.58.3.M#8 7/21/14 10:20 M 1
VeSo
DaBa XXX5.58.3.M#8 7/21/14 16:20 M 1
MiYo XXX5.58.3.M#8 7/22/14 9:25 M 1
JaAy XXX5.58.3.M#8 7/15/14 10:03 M 1
AnSu XXX5.58.3.M#8 7/22/14 11:34 M 1
ThYo
PhMu
ScSa
RiBr
JoVa
TrMa XXX5.58.3.M#8 7/14/14 15:33 M 1
JaSt XXX5.58.3.M#8 7/16/14 11:50 M 1
WaFu XXX5.58.3.M#8 7/16/14 12:11 M 1
Erma XXX5.58.3.M#8 7/21/14 13:31 M 1
JaJa XXX5.58.3.M#8 7/21/14 15:37 M 1
JeSc XXX5.58.3.M#8 7/14/14 9:36 S 1
HoSh
LoWh
JiWh
RiGr
StDa XXX5.58.3.M#8 7/10/2014 9:13 M 1
TrBo XXX5.58.3.M#8 7/10/2014 9:52 M 1
SeLo XXX5.58.3.M#8 7/10/2014 13:37 M 1
ChEs XXX5.58.3.M#8 7/10/2014 14:09 M 1
FrTh
AnSm
JoSm XXX5.58.3.M#8 7/21/14 14:25 M 1
HeBa
BoSm XXX5.58.3.M#8 7/16/14 14:26 M 1
BeSt XXX5.58.3.M#8 7/16/14 17:36 M 1
Total 0 13 3 4
% 0% 65% 15% 20%
Grand Total 20
0%
65%
15%
20%
Breakdown of Performance
Needs Warm-
up
Good
Mod
Fail
Average HR
(BLECS |
BE1.2)
125 | 126
Bias 0.02%
SD of Bias ± 1.9%
MAPE 1.0%
<-15% 0%
-15% to -10% 0%
-10% to -5% 1%
5% to 5% 98%
5% to 10% 1%
10% to 15% 0%
>15% 0%
Average HR
(BLECS |
BE1.2)
125 | 126
Bias 0.4%
SD of Bias ± 3.2%
MAPE 2.6%
<-15% 1%
-15% to -10% 1%
-10% to -5% 2%
-5% to 5% 91%
5% to 10% 2%
10% to 15% 1%
>15% 2%

31. © 2017 Valencell, Inc
Validation Report - representative data sets

32. © 2017 Valencell, Inc
Validation Report - representative data sets

33. © 2017 Valencell, Inc
Validation Report - representative data sets

34. Q&A
Thanks for your time!
If you have further questions or
would like to discuss testing
further, please reach out at
info@valencell.com