Downloaded 17 times
![The industry [and academics] lacks a clear method
of benchmarking sensor accuracy…damaging
reputation of the industry and allowing poor quality
sensors to flood the market and erode prices.
If industry awareness of what constitutes a good fitness
tracker does not improve, the fitness tracking industry
will be a low-quality, low-value industry
ABI Research. (February 19, 2015). Hot Tech Innovators. Retrieved from https://www.abiresearch.com/whitepapers/Hot-Tech-Innovators/](https://image.slidesharecdn.com/bunnandeschbach-seacsm2018-180402125549/85/SEACSM-2018-7-320.jpg)
Uploaded byValencell, Inc
SEACSM 2018
The document discusses evaluating biometric wearables from academic and industry perspectives. It covers current wearable form factors and types of biometrics measured. It notes the lack of clear benchmarking in the industry for sensor accuracy. The academic perspective section outlines considerations for testing products and respecting the industry. Industry drivers of wearables include health, stress reduction and body composition monitoring. Standardization efforts aim to improve accuracy, including standards for steps, heart rate, sleep and other metrics. Validating biometrics requires understanding sensors, suitable benchmarks, proper methodology and analyzing results to ensure accuracy.
More Related Content
More from Valencell, Inc
SEACSM 2018
- 1. Evaluating Biometric Wearables: FromAcademics to Industry Jennifer Bunn, PhD Chris Eschbach, PhD Campbell University Valencell Inc.
- 2. The Plan • Currentwearables and biometrics • Academic perspective • What is driving industry • Standards in evaluating biometric device • Validating biometrics
- 3.
- 4. Many Form Factors EarbudsArmbands Wrist Watches
- 5. Biometrics • What thenumbers mean and why they are important • Activity • Sleep • Health • Stress • Movement context • Heart Rate • Oxygen saturation • Heart Rate Variability • Breath Rate • Galvanic skin response • Blood Pressure • Electroencephalography • Calories • Oxygen consumption (including VO2max)
- 6.
- 7. The industry [andacademics] lacks a clear method of benchmarking sensor accuracy…damaging reputation of the industry and allowing poor quality sensors to flood the market and erode prices. If industry awareness of what constitutes a good fitness tracker does not improve, the fitness tracking industry will be a low-quality, low-value industry ABI Research. (February 19, 2015). Hot Tech Innovators. Retrieved from https://www.abiresearch.com/whitepapers/Hot-Tech-Innovators/
- 8.
- 9. A Few Notesabout the Literature • Selection of products to test • Respecting the industry • Updated hardware and software • Proper wear of the device under test (DUT)
- 10. Research Potpourri • Benchmarkcomparison • Steps – real-time (Sears et al., An et al., Modave et al., Fokkema et al.), pedometer, video monitoring (Chen et al., Tudor-Locke et al.), none (Wen et al.) • Heart rate – ECG (Gillinov et al., Wallen et al., Jo et al.), chest strap (Stahl et al.) • Energy expenditure – metabolic analyzer (Wallen et al., Woodman et al. Chowdhury et al.) • Sampling rate of the metric • Specific time points (Wallen et al., Gillinov et al.), specific steps (Modave et al.), completion of activity (An et al, Sears et al.), continuous (Jo et al.)
- 11. Research Potpourri • Labvs. free-living • Lab (Gillinov et al, Wallen et al., Sears et al.) lacks external validity • Free-living (An et al., Jo et al.) – difficult to test, what is a step? • Acceptable accuracy • 5% (Feito et al.) • 10% (CTA Physical Activity Standard) • ±3-5 bpm (Terbizan et al.) • Appropriate statistics
- 12. Research potpourri Study CorrelationMAPE Test of differences Test of Equivalence Bland- Altman An et al. Chen et al. Chowdhury et al. Fokkema et al. Gillinov et al. Jo et al. Nelson et al. Stahl et al. Wallen et al. Woodman et al.
- 13.
- 14. Base: Online U.S.adults who are physically active (n=932); Online U.S. adults who own a fitness tracker and who are physically active (n=496) Q5. For which of the following reasons, if any, do you exercise? Consumer Electronics Association. (January 6, 2015). Wearable Activity Trackers: Engaging Consumers to Monitor Their Health. Retrieved from http://www.ce.org/research.aspx The needs 4% 10% 9% 11% 5% 20% 14% 30% 41% 36% 42% 43% 50% 51% 58% 68% 10% 13% 14% 18% 18% 24% 27% 51% 56% 65% 66% 67% 69% 71% 79% 85% Part of a rehabilitation program Manage diabetes Manage a chronic disease (other than heart disease… Manage heart disease Train for a race or competition Your doctor or physician recommended/prescribed… To save money on medical costs Prevent disease Maintain current weight Increase endurance Enjoyment Build muscle or increase strength Lose weight or reduce body fat Reduce stress Look better Improve overall health Consumer Motivation to Exercise Fitness tracker owners Online U.S. adults
- 15. The Needs • Whatconsumers want • Health/Stress Reduction/Body Composition* (performance?) • What industry wants • Consumers • What everyone needs • ACCURACY • Health • Physical • Workload context – energy expenditure • Stress • Workload context- exercise and non exercise • Performance • Workload context – training effect • Do we actually have the above? • Depends on who you are asking and what your measuring with
- 16.
- 17. Standardization efforts • TheHealth and Fitness Technology Division of CTA strives to raise awareness of how consumer technologies can help improve health and fitness. • CTA’s Health and Fitness Technology Subcommittee (R6.4) develops standards, recommended practices, and related documentation for consumer health and fitness technology, including fixed, portable and wearable health and fitness devices. • R6.4 WG 1 – Sleep Monitors • R6.4 WG 2 – Physical Activity Monitoring Standards • R6.4 WG 3 – Consumer EEG Data (No Report – On Hiatus) • R6.4 WG 4 - Consumer Stress Monitoring Technologies • R6.4 WG 5 – Mobile Health Applications Others
- 18. Standardization efforts • Steps •Physical Activity Monitoring for Fitness Wearables - Step Counting ANSI/CTA-2056 • Blood pressure • Non-invasive sphygmomanometers — Part 2: Clinical investigation of automated measurement type ANSI/AAMI/ISO 81060-2:2013 • IEEE Standard for Wearable, Cuffless Blood Pressure Measuring Devices IEEE Std 1708-2014 • Heart rate • Physical Activity Monitoring for Heart Rate ANSI/CTA-2065 • ECG • Medical electrical equipment — Part 2-27: Particular requirements for the basic safety and essential performance of electrocardiographic monitoring equipment ANSI/AAMI/IEC 60601-2-27:2011 • Sleep • Definitions and Characteristics forWearable Sleep Monitors CTA-2052.1 • Methodology of Measurements for Features in SleepTracking CTA-2052.2 • In progress • Sleep- ANSI/CTA/NSF-2052.3, Performance Criteria andTesting Protocols for Features inSleepTracking ConsumerTechnology Devices and Applications. • Intensity-ANSI/CTA-2074, Intensity Metrics: Physical Activity Monitoring • Stress- ANSI/CTA-2068, Definitions andCharacteristics of ConsumerStress MonitoringTechnologies • Mobile health-CTA-2073, Guiding Principles of Practice andTransparency for Mobile Health Solutions
- 19.
- 20. Why Validate • BecauseACCURACY is required • During everyday life • Determine moderate and vigorous intensity • Quantify training load • Determine caloric measures for energy balance • Monitor stress levels • Detect changes in “health” • During periods of poor health (clinical or first responder) • Accuracy is a matter of life and death • During gaming • To deliver ideal conditions (fun/stress/emotions)
- 21. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 22. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 23. Validating Biometrics • Knowyour sensors- context matters WORKLOAD CONTEXT
- 24. Validating Biometrics • Knowyour sensors- context matters WORKLOAD CONTEXT
- 25. Stress Testing theBiometric Sensor • Know your sensors • What is the use case
- 26. Stress Testing theBiometric Sensor • Know your sensors • Underlying science • PPG vs ECG
- 27. Stress Testing theBiometric Sensor • Know your sensors • Points of failure • How it fails • Where it fails
- 28. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 29. Stress Testing theBiometric Sensor • Set-up • Data retrieval from devices
- 30. Stress Testing theBiometric Sensor • Set-up • Data retrieval from the DUT
- 31. Stress Testing theBiometric Sensor • Proper methodology • Set-up • Data retrieval from the device under test (DUT)
- 32. Stress Testing theBiometric Sensor • Planning • Documentation • DUT - ABCv.w.x.y#z • ABC-> indicates project • v) Modifications to the mechanics of the core structure (i.e. external id, stalk, sensor angle, ear tip) • w) Additions to the core (gel designs, straps, fin modifications) • x) Modifications to the sensor, sensor components or firmware • y) Indicates sizing • #z) Actual unit number (i.e ABC 5.4.3.M#1 should be the same as ABC 5.4.3.M#5)
- 33. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 34. Stress Testing theBiometric Sensor • Suitable benchmarks • Heart rate • Heart rate variability • Accelerometery • Distance • Breathing rate • Oxygen consumption including VO2max (Calories) • Blood Pressure
- 35. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 36. Stress Testing theBiometric Sensor • Correct pool of participants • Type (age, bmi, gender, skin type) • Number
- 37. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 38. Lifestyle In-Session HealthMonitoring Lifestyle In-Session Health Monitoring Wearability 24/7 comfort; visible Stable during target activities; visible or invisible 24/7; invisible Accuracy “Good enough” for assessments Real-time accuracy critical Real-time accuracy critical Battery Life ≥ 3 days ≥ 3 hours ≥ 1 month Engagement Daily, weekly, & monthly Daily, weekly, & monthly Clinician-dependent Stress Testing the Biometric Sensor • Proper methodology
- 39. Stress Testing theBiometric Sensor • Proper methodology • Protocols for assessment should include the following (regardless of use case): • Rest • Steady • Dynamic • Mode (running, cycling, strength, lifestyle) • Environmental conditions
- 40. Stress Testing theBiometric Sensor • Proper methodology • Protocols for assessment • Environmental conditions • Heat, cold, sunlight
- 41. Validating Biometrics • Whyvalidate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
- 42. Stress Testing theBiometric Sensor • Considerations • All “benchmark units” have potential problems. • Synchronize the data start time (because of latency or lack of sync)
- 43. Stress Testing theBiometric Sensor • Sound and consistent evaluation techniques • Qualitative and quantitative measures • Subjective scoring • Quantitative scoring • Mean absolute percent error • Distribution data • Correlation • Equivalence testing • Mean bias • Data availability • Latency
- 44. Best practices • Standardization •Participants • Settings • Use cases • Benchmark devices • Data dropout