Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective

565 views

Published on

by Dr. Kiran Marri, Vice President - Digital Engineering Services & James Mathew, Senior Consultant - Digital Engineering Services, CSS Corp Limited at STeP-IN SUMMIT 2018 - 15th International Conference on Software Testing on August 30, 2018 at Taj, MG Road, Bengaluru

Published in: Technology
  • Be the first to comment

  • Be the first to like this

How accurate are the Wearable fitness tracker showing 10000 steps in a day: A Testers perspective

  1. 1. © CSS Corp | Confidential | www.csscorp.com 1 Customer Engagement Reimagined How accurate are the wearable fitness trackers for step counting(10000 steps a day): A tester’s perspective James Mathew, Karthikeyan Swaminathan and Dr. Kiran Marri CSS Corp Solutions
  2. 2. © CSS Corp | Confidential | www.csscorp.com 2 Customer Engagement Reimagined Disclaimer This research work, data and results is not intended to substitute for informed medical advice or fitness regime One must not use this information to check, confirm, diagnose or treat any health problem or fitness conditions. Always check with your doctor before altering fitness regime or starting any new fitness routine Several popular brands are tested in this research work, and thereby these brand names are intentionally not mentioned in the paper
  3. 3. © CSS Corp | Confidential | www.csscorp.com 3 Customer Engagement Reimagined Table of Contents Background - wearables, types and fitness band Story of 10000 steps Learnings, Conclusions Thinking like a tester - Scenarios, Design and Wearable Test Framework Key results Automation strategies
  4. 4. © CSS Corp | Confidential | www.csscorp.com 4 Customer Engagement Reimagined 71% in age group 16 to 24 want “wearable tech,” - Global Web Index  https://www.grandviewresearch.c om/industry-analysis/wearable- technology-market  https://www.statista.com/chart/33 70/wearable-device-forecast/ Rapid growth in wearables market Wrist wear - most popular wearable 2011: Fitbit Ultra (2011) 2012: Nike FuelBand, Fitbit One, Sony SmartWatch 2013: Pebble, Fitbit Flex, Sony 2, Samsung Gear, Nike FuelBand SE, MI Band 2014: Fitbit Force, Samsung Gear 2, LG Watch, Motorola watch, Garmin 2015: Apple watch Popular products in the wrist wear category Landscape of wearables Market size | Types of wearables | Popular products | Sensors  UV sensor  GPS | Gyro  Ambient light sensor  3 axis accelerometer  Skin temperature sensor  Galvanic skin sensor  Haptic vibration motor Wrist wear: Microsoft Band 2 sensors With 300% growth, wrist wear (band, watch) is the most popular device in the market 3 4 1 2
  5. 5. © CSS Corp | Confidential | www.csscorp.com 5 Customer Engagement Reimagined Wrist (fitness) band Purpose | Measurements | Experience | Why 10000 steps 32 Experiences of wearable like a Fitbit, Garmin watch, or Apple watch Parameters measuredWhy fitness band?1  Activity tracking  Track progress on different days  Focus on training/goal-oriented  Habit information  Brand-style quotient  Weight loss programs  Calorie burn counter  Step counter/tracker  Time trends in different segments  Distance tracker  Speed of pace in segments  Infographics  Personalized activity tracking is the most common use of wrist band  Burning 300 calories is recommended by fitness community to lose weight, and this translates to 10000 steps (30% at brisk pace, 2500 steps is 1 mile approximately) 10000 steps a day in urban group, and over 90% enjoy using wrist bands
  6. 6. © CSS Corp | Confidential | www.csscorp.com 6 Customer Engagement Reimagined Tester’s perspective: Fitness band measurement | Hypothesis driven approach Is the data displayed by the fitness band accurate?1 Representative Test Hypothesis Influence of arm swing on the band readings 2 Effect of high activity vs low activity vs stationary position 3 Indoor (treadmill) vs external (outdoor) for different step count 4 Influence of person’s height on the step count 5 Problem statement: How accurate is the band, functionally? High end wrist band vs low cost wrist band vs smart watch 6 Type of walking (regular vs running vs brisk walking) 7 Functional | Performance | Security | Usability | Localization | Network connectivity | Rigidity | Battery | Storage capacity | Luminous | Data integrity | Reports | Compatibility | Environment Coverage GROUP A GROUP B GROUP D GROUP C Test Scenario Design : Activities with Steps | Stationary | Individuality | Fitness brand internals
  7. 7. © CSS Corp | Confidential | www.csscorp.com 7 Customer Engagement Reimagined Test considerations:  Demography of the wearables: Typical user groups identification such as runners, swimmers etc. across geos  Pre-condition: Definition of ordinary and extra-ordinary usage of the wearable  Post-condition: Defects could relate to the hardware (such as rigidity), software (such as algorithm fine-tuning) or configuration (such as calorie re-calibration)  Scope of testing: device black box functional QA. Test Design Phase Scenario design & define Hypothesis Analysis (factors, attributes Planning Reviews Servo automation Manual Requirements Analysis Wearables Test Strategy Test Design Test Execution Story Demo/Acceptance Test Execution Phase Test reports Release Retrospection Proposed elements of wearable testing methodology driven by Hypothesis test approach Independent accelerometer verification Image detection Test Scenario Design : Activities with Steps | Stationary | Individuality | Fitness brand internals
  8. 8. © CSS Corp | Confidential | www.csscorp.com 8 Customer Engagement Reimagined 1 STEP Heel Off Toe OffHeel Strike Flat foot Mid stance Mid swing Definition of a step: A typical step and arm swing
  9. 9. © CSS Corp | Confidential | www.csscorp.com 9 Customer Engagement Reimagined Scope Measurement Automation Approaches/Tools Outdoor walking + Automation Accelerometer Indoor walking + Automation Video/ Object tracking Arm swings + Automation Video/ Object tracking Stationary (Non activity) Manual - Stationary (In motion) Manual - Height influence + Automation Video/ Object tracking Distance walking + Automation Simulation Time bound walking + Automation Simulation Environment/ Surface Manual - Test Scenario Design Automation strategies Is the data displayed from the fitness ….1 Influence of arm swing on the band ….. 2 Effect of high activity vs stationary …. 3 Indoor (treadmill) vs external (outdoor) …. 4 Influence of person height on the step …. 5 High end wrist band vs low cost wrist …… 6 Type of walking …….. 7
  10. 10. © CSS Corp | Confidential | www.csscorp.com 10 Customer Engagement Reimagined Study Details: Subjects | Fitness Bands | Distances | Steps Parameter Details and information Specifics for the results # of participants 20-30 adults over a period of 12 months 22 healthy adults# Gender All 3 Females | 19 Males Age group Tested on Geriatrics, Gen X, Gen Y 23 to 50 Height of subjects 153 to 181cm # of brand tested 6 brands, discarded 2 (too few samples or errors) 4 popular brands Concurrency test 1 brand x 3 1 brand Fixed steps test Factors: Practical, Manageable 40 to 500 steps Fixed distance test 100m to 5000m 100m to 400m Arm swing Normal, and up to elbow levels elbow levels benchmark Speed of walk | run Individual capability None # Results carried from Mar 2018 is included in the study
  11. 11. © CSS Corp | Confidential | www.csscorp.com 11 Customer Engagement Reimagined Steps Walked Counting the steps walked Accelerometer variation during walking  Manual counting of steps during testing can be cumbersome and error prone  A mobile app which measures the accelerometer values is used  Z axis indicates direction of move  X axis indicates the left to right movement  A combination of readings from X and Z axis accelerometer yields an accurate count of steps -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141 148 155 162 169 176 183 190 197 Accelerometer variation X axis Z axis Automation : Motion tracking of steps for indoor & outdoor An mobile app was build to log the recordings, and # of steps derived
  12. 12. © CSS Corp | Confidential | www.csscorp.com 12 Customer Engagement Reimagined Reference: https://physlets.org/tracker/ Gait | Speed | Arm length | Swing Motion Video Feature analysis Feature selection Feature vectors Model Frames Feature extraction Step detection Feature detection: Machine learning Object Tracking Software Automation : Motion tracking of indoor events Object tracking software is simple and easy but can be used only for treadmill with reference
  13. 13. © CSS Corp | Confidential | www.csscorp.com 13 Customer Engagement Reimagined Simulation  Changing gear while driving a car: With the vertical arm orientation, testing for a person changing gear of car can be done  Writing activity: With a horizontal arm orientation, testing for a person writing can be done  Activities such as steering wheel simulation, writing, physical exercise can be simulated Simulation  Short and long arm influence can be tested by changing the arm length  Fast and slow swing variations in gait can be replicated Servo motor Device Arm Servo motor Device Arm Configuration: The device and arm are hanging from the servo motor Configuration: The device and arm are held up from the servo motor Hand movement | Arm length | Swing Prototype model is developed for testing durability* *Results not included in the paper Automation : Simulation of arm swing
  14. 14. © CSS Corp | Confidential | www.csscorp.com 14 Customer Engagement Reimagined Results
  15. 15. © CSS Corp | Confidential | www.csscorp.com 15 Customer Engagement Reimagined Hypothesis: Fixed distance walk results in similar variations for the band (if any) Result 1. Consistency check of the Fitness band Subjects are requested to walk casually for a fixed distance | Two brands are experimented | Outdoors %variations Unique variations 1. Data spread observed in both positive and negative variation (-16% to +8%) for normal walking conditions 2. Negative variations are stronger in both the brands 3. Indoor (on the treadmill) showed fewer variations as compared to outdoor KEY OBSERVATIONS AND ANALYSIS Result: Fail | Fitness bands tend to have inconsistent performance over 10+ walks TEST SCENARIO TESTDATA&PLOTS
  16. 16. © CSS Corp | Confidential | www.csscorp.com 16 Customer Engagement Reimagined Hypothesis: Speed of walking must notinfluence the fitness band output Result 2. Effect of speed (brisk vs normal walking) Subjects are requested to walk at normal and brisk speed for a fixed distance | Activity conducted both at outdoors and indoors 1. Fast walking had more spread as compared to slow walking 2. Spikes of inconsistency observed in normal walking and not reproducible 3. Statistically the hypothesis can be rejected as p-value is 0.06 (for 0.05) but due to F and F-crit value shows otherwise KEY OBSERVATIONS AND ANALYSIS Result: Fail | Fitness band outcome may have strong influence on the speed of walking ANOVA Source of Variation SS df MS F p-value F crit Between Groups 90.38 1 90.38 3.78 0.06 4.08 Within Groups 957.24 40 23.93 Total 1047.62 41 Reference: http://www.statisticshowto.com/probability-and-statistics/f-statistic-value-test/ FAST NORMAL TEST SCENARIO TESTDATA&PLOTS
  17. 17. © CSS Corp | Confidential | www.csscorp.com 17 Customer Engagement Reimagined Hypothesis: With increase or decrease in steps, consistency is maintained Result 3. Effect of multiple steps KEY OBSERVATIONS AND ANALYSIS Result: Fail | Fitness band accuracy reduces with steps, and marginally more in indoors Outdoor Indoor TEST SCENARIO TESTDATA&PLOTS Subjects are requested to walk fixed steps or distance | Natural speed | outdoors and indoors 1. Trends show that the error deviation increases with step count 2. The analysis is performed for over 100 steps 3. No statistical significance between outdoors and indoor data, with indoor showing higher COV
  18. 18. © CSS Corp | Confidential | www.csscorp.com 18 Customer Engagement Reimagined Hypothesis: In Idle state, steps do not add up Result 4. Stationary position performance Subjects in stationary position for fixed time, and requested to perform movements such as writing, eating, typing | Multiple brands | IndoorsTEST SCENARIO 1. Most of the brands gave at least 30% of the data accurately, with the highest being 70% 2. The average deviation ranged from 2% to 17%, and standing and sitting stationary movements, gave similar results 3. Accuracy of the steps is specific to the brand KEY OBSERVATIONS AND ANALYSIS Result: Fail | Error specific to Fitness band, but accurate at least 30% of times TESTDATA&PLOTS In motion (driving car| commute) 1. Error in the range of 10% to 100% 2. Data tested on different brands with fewer samples, and hence results are non-conclusive
  19. 19. © CSS Corp | Confidential | www.csscorp.com 19 Customer Engagement Reimagined Hypothesis: Arm swing must not have any influence on Fitness band accuracy Result 5. Effect of arm swing Subjects are requested to walk with arm swing, normal and forcefully high for 100 steps | Activity conducted both outdoors and indoorsTEST SCENARIO 1. Few brands had minimum influence; certain brands had statistically significant difference with arm swing 2. Maximum and min variation between 2% and 14% 3. Overall analysis did NOT show any statistical difference, and hence this issue is very specific to brand KEY OBSERVATIONS AND ANALYSIS Result: Fail | Fitness band accuracy has influence on arm swing and very specific to the brand SHORT LONG TESTDATA&PLOTS
  20. 20. © CSS Corp | Confidential | www.csscorp.com 20 Customer Engagement Reimagined Hypothesis: Fitness band when not calibrated for height results in error Result 6. Influence of Height (UNCALIBRATED)TESTDATA&PLOTS 1. Height of the individual is important in the step count and concurred with Industry observations# 2. Actual steps matched closer with industry data for taller people as compared to shorter people 3. Fitness band accuracy is better for taller people (175+ cm) as compared to shorter people, due to difference in strides KEY OBSERVATIONS AND ANALYSIS Result: Pass | Fitness bands appear to be influenced by person height # Reference: https://www.verywellfit.com/how-many-walking-steps-are-in-a-mile-3435916 #ofSteps Height (cm) Subjects are requested to walk casually for a fixed distance | Calibrated for 175cm | Activity conducted outdoorsTEST SCENARIO
  21. 21. © CSS Corp | Confidential | www.csscorp.com 21 Customer Engagement Reimagined 1. The main objective of this study was to bring out the factors that influence fitness devices (wearables) 2. Wearable devices testing framework designed using Hypothesis testing, and 3 automation methods have been proposed, and tested on 5 popular brands 3. The study brings out the importance of “individual parameters” (height, stride style, gait and arm swings) that can alter the accuracy of the step count 4. Dominant activities such as walking are prone to lesser errors, but many stationary activities involving hand movement lead to 50% or more issues 5. The study has a significant importance in rehabilitation and geriatrics, where the day long “steps” are taken into account. This testing approach can be applied to validate other wearables in the field of medical research, sports mechanics and animal management 6. The study proves that principles of testing can be enhanced using analytics, IOT, imaging and automation Conclusions Summary | Learnings | Future plans
  22. 22. © CSS Corp | Confidential | www.csscorp.com 22 Customer Engagement Reimagined How accurate are wearables as step-counters?(target of 10000 steps a day) Results show that dependency factors can skew the step count (Non-walk/run activities: Stationary Cycling, Car driving, Commute, hand movements involved during eating, writing and more !!) Recommendations 1. Decide when to wear (Wear fitness tracker only during active walk/ running) 2. Calibrate before use(brands that we tested have similar issues on different scales) 3. All day users (6AM to 10PM) to reset target as 12000-14000 steps a day, to meet the goal Conclusions Inferences| Recommendations
  23. 23. © CSS Corp | Confidential | www.csscorp.com 23 Customer Engagement Reimagined Q & A Comments: Dr. Kiran Marri : Kiran.Marri@csscorp.com James Mathew : James.Mathew@csscorp.com
  24. 24. © CSS Corp | Confidential | www.csscorp.com 24 Customer Engagement Reimagined Appendix
  25. 25. © CSS Corp | Confidential | www.csscorp.com 25 Customer Engagement Reimagined Results reporting QA focus Test activities considered • Mobile activities • Cycling • Driving a car • Travelling in a bus • Immobile activities • Typing/writing • Exercise such as dumb bells • Idling activities • Sitting in a chair • Sleeping in a bed • Travelling in a car • Typical walking • Outdoor walking (slow or fast) • Indoor walking (treadmill based) Test subjects • 16 people • 3 brands of fitness wearables • Both genders (3 females, 13 males) • Age group: 23 to 50 • Testing done over 1 year (data taken in last 5 months are represented • Over week-days and week-ends Factors considered • Arm length variations • Arm swing angle and speed • Fitness tracker on dominant vs. non-dominant hand • Mobile vs. immobile activity • Wearing it in hand vs. ankle
  26. 26. © CSS Corp | Confidential | www.csscorp.com 26 Customer Engagement Reimagined Results reporting …continued QA observations Row Labels Average of Deviation (%) In motion with no physical activity 35% In motion with physical activity -1% In stationary, with dominant hand active 55% In stationary, with non- dominant hand active 0% Date Tester ID Activity ID Test condition Test steps Expected results (steps) Actual results Observations -Deviation (%) 27-May-18 ABC01 A02 Human: Subject is involved in slow/fast walking activity Tester walks 100 steps exactly. The number of steps before and after the travel are read from the fitness tracker 100 99 1% 27-May-18 ABC01 A02 Human: Subject is involved in slow/fast walking activity Tester walks 100 steps exactly. The number of steps before and after the travel are read from the fitness tracker 100 105 -5% 27-May-18 ABC01 A03 Human: Subject is involved in performing a regular activity such as eating, typing Tester sits in a chair and performs a regular activity such as eating, typing, after wearing the fitness tracker in the dominant hand. The number of steps before and after performing the activity are noted 0 0 0% 01-Jul-17 ABC02 A01 Human: Subject is involved in indoor- cycling Tester wears the fitness tracker in ankle. Uses an indoor cycling exercise machine. Makes 100 times of circular motion. Stops the cycle. The number of steps before and after the exercise are noted 0 100 100% 01-Jul-17 ABC02 A04 Human: Subject is just being idle Tester sits in a chair for 30 minutes in a easy position, with causal movements. The number of steps before and after this positioning are noted 0 0 0% 25-May-18 ABC03 A02 Human: Subject is involved in slow/fast walking activity Tester walks 100 steps exactly. The number of steps before and after the travel are read from the fitness tracker 100 112 -12% 28-May-18 ABC04 A02 Human: Subject is involved in slow/fast walking activity Tester walks 100 steps exactly. The number of steps before and after the travel are read from the fitness tracker 100 99 1% -10% 0% 10% 20% 30% 40% 50% 60% In motion with no physical activity In motion with physical activity In stationary, with dominant hand active In stationary, with non- dominant hand active Step count deviation chart
  27. 27. © CSS Corp | Confidential | www.csscorp.com 27 Customer Engagement Reimagined RoI analysis of test harness Objective: Measure step count on fitness tracker with indoor slow/fast walking using different subjects Inference: Measuring step count on fitness tracker with indoor walking, is optimal to get tested in a servo based test harness rather than using different personnel subjects, as we get 14% from the 1st test iteration Cost (USD) Savings (USD) Remarks Infrastructure, Lab Equipment(s) 2,500 Cycle time reduction 2,500 (10 USD per person x 250 person) Early defects 11,750 (1000 USD per complex defect, 750 USD per medium defect to 500 USD per simple defect; Complex - 5; Effort (Research, Training, POC) 10,000 Total 12,500 Total 14,250 ROI = 14%

×