Getting Roi Numbers From Ut Des Moines10 Sept09

645 views

Published on

How to make usability more meaningful to your business measures

  • Be the first to comment

  • Be the first to like this

Getting Roi Numbers From Ut Des Moines10 Sept09

  1. 1. Getting ROI Numbers from Usability Testing John Sorflaten , PhD Certified Professional Ergonomist (CPE) Certified Usability Analyst (CUA) “ Midwife to successful web and application design” User testing, analysis, strategy, eCommerce, mobile apps User Experience Works http://www.linkedin.com/in/johnsorflaten (see SlideShare) www.userexperienceworks.com www.iphoneroi.com 641-455-1873
  2. 2. Overview <ul><li>Making usability more meaningful and consumable to business… </li></ul><ul><li>Quantitative usability testing (UT) </li></ul><ul><li>Converting UT results to business measures </li></ul><ul><li>Presenting UT results in relevant business terms </li></ul><ul><li>Why scare you with “Quantitative” word? </li></ul>
  3. 3. 1. Beyond “More” and “Less” <ul><li>Measure usability to learn “quantity” of… </li></ul><ul><ul><li>Efficiency (time-on-task) </li></ul></ul><ul><ul><li>Effectiveness (task success rates) </li></ul></ul><ul><ul><li>Satisfaction (subjective ratings) </li></ul></ul><ul><ul><li>Learning (time to learn) </li></ul></ul><ul><ul><li>Errors (ouch) </li></ul></ul><ul><li>“ Quantity” allows ROI calculations (!) </li></ul>
  4. 4. Use these free ROI calculators… <ul><li>http://www.humanfactors.com/downloads/roi.asp </li></ul>
  5. 5. ROI Calculator — Increased Productivity… <ul><li>http://www.humanfactors.com/training/roi_incr_prodvt.htm </li></ul>
  6. 6. ROI Calculator — Increased Productivity… <ul><li>http://www.humanfactors.com/training/roi_incr_prodvt.htm </li></ul>
  7. 7. 2. Remember:“Margin of Error” <ul><li>Like voting polls… measures are fuzzy. Remember +/- 3%? (You need 1067 people needed to get +/-3%…) </li></ul><ul><li>If 8 of 10 test participants succeed, DON’T say “80% passed.” </li></ul><ul><li>Best you can say: 48% to 96% passed. </li></ul><ul><li>www.measuringusability.com/wald.htm </li></ul>
  8. 8. 3. Use “Time-on-Task” for ROI <ul><li>Reduces margin-of-error for ROI calcs. </li></ul><ul><li>Use “geometric mean” not “average” (just like “median house price”) </li></ul><ul><li>What’s your real target for time savings? (test) </li></ul>http://www.measuringusability.com/time_intervals.php
  9. 9. 4. Go for the “lower bound”… <ul><li>www.humanfactors.com/downloads/quantitative-usability </li></ul>Finally, success…
  10. 10. Converting UT Results to Business Measures <ul><li>Efficiency (time-on-task) (DONE!) </li></ul><ul><li>Effectiveness (task success rates) </li></ul><ul><li>Satisfaction (subjective ratings) </li></ul><ul><li>Learning (time to learn) </li></ul><ul><li>Errors (ouch) </li></ul>
  11. 11. 1. Effectiveness (“Pass” Rate) <ul><li>www.humanfactors.com/downloads/quantitative-usability </li></ul>No real difference….
  12. 12. Need spectacular contrasts <ul><li>http://www.measuringusability.com/comp_sample.php </li></ul>
  13. 13. Because these are — ”samples” <ul><li>www.humanfactors.com/downloads/quantitative-usability </li></ul>No real difference….
  14. 14. Better: compare with “population” <ul><li>http://www.measuringusability.com/comp_sample.php </li></ul>A real difference….
  15. 15. What’s the best we can say? <ul><li>http://www.measuringusability.com/comp_sample.php </li></ul>Using what criteria? 80%? 90%….
  16. 16. What’s the best we can say? <ul><li>http://www.measuringusability.com/comp_sample.php </li></ul>Using what criteria? 80%? 90%….
  17. 17. If high stakes: U need larger “N” <ul><li>www.humanfactors.com/download/quantitative-usability </li></ul>Your benchmark: 85% will succeed on task…
  18. 18. ROI = “profit only if usable by at least 85% of people who try…”
  19. 19. Converting UT Results to Business Measures <ul><li>Efficiency (time-on-task) (DONE!) </li></ul><ul><li>Effectiveness (task success rates) (DONE) </li></ul><ul><li>Satisfaction (subjective ratings) </li></ul><ul><li>Learning (time to learn) </li></ul><ul><li>Errors (ouch) </li></ul>
  20. 20. 3. Satisfaction (>80% = goal) <ul><li>http://measuringuserexperience.com/ </li></ul>116/129 => 90 th Percentile scores of 81.2+ = 90 th Percentile
  21. 21. 3. Satisfaction (goal: >80%) <ul><li>http://en.wikipedia.org/wiki/System_Usability_Scale </li></ul>
  22. 22. 3. Satisfaction (>80% = goal) <ul><li>http://www.upassoc.org/usability_resources/conference/2004/UPA-2004-TullisStetson.pdf </li></ul>
  23. 23. 3. Satisfaction (>80% = goal) <ul><li>http://www.upassoc.org/usability_resources/conference/2004/UPA-2004-TullisStetson.pdf </li></ul>
  24. 24. 3. Satisfaction (>80% = goal) <ul><li>http://www.upassoc.org/usability_resources/conference/2004/UPA-2004-TullisStetson.pdf </li></ul>
  25. 25. 3. Satisfaction (>80% = goal) <ul><li>http://www.upassoc.org/usability_resources/conference/2004/UPA-2004-TullisStetson.pdf </li></ul>
  26. 26. Final Report: Satisfaction <ul><li>The test of the High Point Quote application with 18 participants (and 17 results on the SUS) resulted in an average score of 79.7. </li></ul><ul><li>This compares very well with the 81.2 score that represents the 90 th percentile. We conclude that the application created a very good feeling of “usability” in our test participants. </li></ul>
  27. 27. Final Report: Success Rate <ul><li>All 18 participants successfully completing the main task. </li></ul><ul><li>This implies that we can have 97.32% confidence of meeting our target that 85% of people like our participants will succeed. </li></ul><ul><li>We can also say that we have a 95% chance that at least 88% can do the task. </li></ul>http://www.measuringusability.com/onep.php
  28. 28. Converting UT Results to Business Measures <ul><li>Efficiency (time-on-task) (DONE!) </li></ul><ul><li>Effectiveness (task success rates) (DONE) </li></ul><ul><li>Satisfaction (subjective ratings) (DONE) </li></ul><ul><li>Learning (time to learn) (See time-on-task) </li></ul><ul><li>Errors (See time-on-task) </li></ul>
  29. 29. UT Results in Business Terms <ul><li>What percent of visitors can do the task? </li></ul><ul><li>Our income depends on your prediction! </li></ul><ul><li>Use estimates like…. </li></ul><ul><ul><li>95% confidence that 88% of end-users can do it! </li></ul></ul><ul><ul><li>60% chance that 85% can do it (uhmmmm) </li></ul></ul><ul><ul><li>95% chance that 60% of end-users can do it… </li></ul></ul><ul><ul><li>! Goforit… </li></ul></ul><ul><li>E-mail your ideas: john@userexperienceworks.com </li></ul>

×