Successfully reported this slideshow.
Your SlideShare is downloading. ×

SEACSM 2018

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 44 Ad

More Related Content

Similar to SEACSM 2018 (20)

More from Valencell, Inc (20)

Advertisement

Recently uploaded (20)

SEACSM 2018

  1. 1. Evaluating Biometric Wearables: From Academics to Industry Jennifer Bunn, PhD Chris Eschbach, PhD Campbell University Valencell Inc.
  2. 2. The Plan • Current wearables and biometrics • Academic perspective • What is driving industry • Standards in evaluating biometric device • Validating biometrics
  3. 3. Wearables
  4. 4. Many Form Factors Earbuds Armbands Wrist Watches
  5. 5. Biometrics • What the numbers mean and why they are important • Activity • Sleep • Health • Stress • Movement context • Heart Rate • Oxygen saturation • Heart Rate Variability • Breath Rate • Galvanic skin response • Blood Pressure • Electroencephalography • Calories • Oxygen consumption (including VO2max)
  6. 6. The Problem
  7. 7. The industry [and academics] lacks a clear method of benchmarking sensor accuracy…damaging reputation of the industry and allowing poor quality sensors to flood the market and erode prices. If industry awareness of what constitutes a good fitness tracker does not improve, the fitness tracking industry will be a low-quality, low-value industry ABI Research. (February 19, 2015). Hot Tech Innovators. Retrieved from https://www.abiresearch.com/whitepapers/Hot-Tech-Innovators/
  8. 8. Academic perspective
  9. 9. A Few Notes about the Literature • Selection of products to test • Respecting the industry • Updated hardware and software • Proper wear of the device under test (DUT)
  10. 10. Research Potpourri • Benchmark comparison • Steps – real-time (Sears et al., An et al., Modave et al., Fokkema et al.), pedometer, video monitoring (Chen et al., Tudor-Locke et al.), none (Wen et al.) • Heart rate – ECG (Gillinov et al., Wallen et al., Jo et al.), chest strap (Stahl et al.) • Energy expenditure – metabolic analyzer (Wallen et al., Woodman et al. Chowdhury et al.) • Sampling rate of the metric • Specific time points (Wallen et al., Gillinov et al.), specific steps (Modave et al.), completion of activity (An et al, Sears et al.), continuous (Jo et al.)
  11. 11. Research Potpourri • Lab vs. free-living • Lab (Gillinov et al, Wallen et al., Sears et al.) lacks external validity • Free-living (An et al., Jo et al.) – difficult to test, what is a step? • Acceptable accuracy • 5% (Feito et al.) • 10% (CTA Physical Activity Standard) • ±3-5 bpm (Terbizan et al.) • Appropriate statistics
  12. 12. Research potpourri Study Correlation MAPE Test of differences Test of Equivalence Bland- Altman An et al. Chen et al. Chowdhury et al. Fokkema et al. Gillinov et al. Jo et al. Nelson et al. Stahl et al. Wallen et al. Woodman et al.
  13. 13. Industry drivers
  14. 14. Base: Online U.S. adults who are physically active (n=932); Online U.S. adults who own a fitness tracker and who are physically active (n=496) Q5. For which of the following reasons, if any, do you exercise? Consumer Electronics Association. (January 6, 2015). Wearable Activity Trackers: Engaging Consumers to Monitor Their Health. Retrieved from http://www.ce.org/research.aspx The needs 4% 10% 9% 11% 5% 20% 14% 30% 41% 36% 42% 43% 50% 51% 58% 68% 10% 13% 14% 18% 18% 24% 27% 51% 56% 65% 66% 67% 69% 71% 79% 85% Part of a rehabilitation program Manage diabetes Manage a chronic disease (other than heart disease… Manage heart disease Train for a race or competition Your doctor or physician recommended/prescribed… To save money on medical costs Prevent disease Maintain current weight Increase endurance Enjoyment Build muscle or increase strength Lose weight or reduce body fat Reduce stress Look better Improve overall health Consumer Motivation to Exercise Fitness tracker owners Online U.S. adults
  15. 15. The Needs • What consumers want • Health/Stress Reduction/Body Composition* (performance?) • What industry wants • Consumers • What everyone needs • ACCURACY • Health • Physical • Workload context – energy expenditure • Stress • Workload context- exercise and non exercise • Performance • Workload context – training effect • Do we actually have the above? • Depends on who you are asking and what your measuring with
  16. 16. Standardization
  17. 17. Standardization efforts • The Health and Fitness Technology Division of CTA strives to raise awareness of how consumer technologies can help improve health and fitness. • CTA’s Health and Fitness Technology Subcommittee (R6.4) develops standards, recommended practices, and related documentation for consumer health and fitness technology, including fixed, portable and wearable health and fitness devices. • R6.4 WG 1 – Sleep Monitors • R6.4 WG 2 – Physical Activity Monitoring Standards • R6.4 WG 3 – Consumer EEG Data (No Report – On Hiatus) • R6.4 WG 4 - Consumer Stress Monitoring Technologies • R6.4 WG 5 – Mobile Health Applications Others
  18. 18. Standardization efforts • Steps • Physical Activity Monitoring for Fitness Wearables - Step Counting ANSI/CTA-2056 • Blood pressure • Non-invasive sphygmomanometers — Part 2: Clinical investigation of automated measurement type ANSI/AAMI/ISO 81060-2:2013 • IEEE Standard for Wearable, Cuffless Blood Pressure Measuring Devices IEEE Std 1708-2014 • Heart rate • Physical Activity Monitoring for Heart Rate ANSI/CTA-2065 • ECG • Medical electrical equipment — Part 2-27: Particular requirements for the basic safety and essential performance of electrocardiographic monitoring equipment ANSI/AAMI/IEC 60601-2-27:2011 • Sleep • Definitions and Characteristics forWearable Sleep Monitors CTA-2052.1 • Methodology of Measurements for Features in SleepTracking CTA-2052.2 • In progress • Sleep- ANSI/CTA/NSF-2052.3, Performance Criteria andTesting Protocols for Features inSleepTracking ConsumerTechnology Devices and Applications. • Intensity-ANSI/CTA-2074, Intensity Metrics: Physical Activity Monitoring • Stress- ANSI/CTA-2068, Definitions andCharacteristics of ConsumerStress MonitoringTechnologies • Mobile health-CTA-2073, Guiding Principles of Practice andTransparency for Mobile Health Solutions
  19. 19. Validation
  20. 20. Why Validate • Because ACCURACY is required • During everyday life • Determine moderate and vigorous intensity • Quantify training load • Determine caloric measures for energy balance • Monitor stress levels • Detect changes in “health” • During periods of poor health (clinical or first responder) • Accuracy is a matter of life and death • During gaming • To deliver ideal conditions (fun/stress/emotions)
  21. 21. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  22. 22. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  23. 23. Validating Biometrics • Know your sensors- context matters WORKLOAD CONTEXT
  24. 24. Validating Biometrics • Know your sensors- context matters WORKLOAD CONTEXT
  25. 25. Stress Testing the Biometric Sensor • Know your sensors • What is the use case
  26. 26. Stress Testing the Biometric Sensor • Know your sensors • Underlying science • PPG vs ECG
  27. 27. Stress Testing the Biometric Sensor • Know your sensors • Points of failure • How it fails • Where it fails
  28. 28. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  29. 29. Stress Testing the Biometric Sensor • Set-up • Data retrieval from devices
  30. 30. Stress Testing the Biometric Sensor • Set-up • Data retrieval from the DUT
  31. 31. Stress Testing the Biometric Sensor • Proper methodology • Set-up • Data retrieval from the device under test (DUT)
  32. 32. Stress Testing the Biometric Sensor • Planning • Documentation • DUT - ABCv.w.x.y#z • ABC-> indicates project • v) Modifications to the mechanics of the core structure (i.e. external id, stalk, sensor angle, ear tip) • w) Additions to the core (gel designs, straps, fin modifications) • x) Modifications to the sensor, sensor components or firmware • y) Indicates sizing • #z) Actual unit number (i.e ABC 5.4.3.M#1 should be the same as ABC 5.4.3.M#5)
  33. 33. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  34. 34. Stress Testing the Biometric Sensor • Suitable benchmarks • Heart rate • Heart rate variability • Accelerometery • Distance • Breathing rate • Oxygen consumption including VO2max (Calories) • Blood Pressure
  35. 35. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  36. 36. Stress Testing the Biometric Sensor • Correct pool of participants • Type (age, bmi, gender, skin type) • Number
  37. 37. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  38. 38. Lifestyle In-Session Health Monitoring Lifestyle In-Session Health Monitoring Wearability 24/7 comfort; visible Stable during target activities; visible or invisible 24/7; invisible Accuracy “Good enough” for assessments Real-time accuracy critical Real-time accuracy critical Battery Life ≥ 3 days ≥ 3 hours ≥ 1 month Engagement Daily, weekly, & monthly Daily, weekly, & monthly Clinician-dependent Stress Testing the Biometric Sensor • Proper methodology
  39. 39. Stress Testing the Biometric Sensor • Proper methodology • Protocols for assessment should include the following (regardless of use case): • Rest • Steady • Dynamic • Mode (running, cycling, strength, lifestyle) • Environmental conditions
  40. 40. Stress Testing the Biometric Sensor • Proper methodology • Protocols for assessment • Environmental conditions • Heat, cold, sunlight
  41. 41. Validating Biometrics • Why validate • Stress testing the biometric sensor • Know your sensors • Plan for assessment • Suitable benchmarks • Correct pool of participants • Proper methodology • Analysis
  42. 42. Stress Testing the Biometric Sensor • Considerations • All “benchmark units” have potential problems. • Synchronize the data start time (because of latency or lack of sync)
  43. 43. Stress Testing the Biometric Sensor • Sound and consistent evaluation techniques • Qualitative and quantitative measures • Subjective scoring • Quantitative scoring • Mean absolute percent error • Distribution data • Correlation • Equivalence testing • Mean bias • Data availability • Latency
  44. 44. Best practices • Standardization • Participants • Settings • Use cases • Benchmark devices • Data dropout

×