Getting Roi Numbers From Ut Des Moines10 Sept09

Getting ROI Numbers from Usability Testing John Sorflaten , PhD Certified Professional Ergonomist (CPE) Certified Usability Analyst (CUA) “ Midwife to successful web and application design” User testing, analysis, strategy, eCommerce, mobile apps User Experience Works http://www.linkedin.com/in/johnsorflaten (see SlideShare) www.userexperienceworks.com www.iphoneroi.com 641-455-1873

Overview
Making usability more meaningful and consumable to business…
Quantitative usability testing (UT)
Converting UT results to business measures
Presenting UT results in relevant business terms
Why scare you with “Quantitative” word?

1. Beyond “More” and “Less”
Measure usability to learn “quantity” of…
Efficiency (time-on-task)
Effectiveness (task success rates)
Satisfaction (subjective ratings)
Learning (time to learn)
Errors (ouch)
“ Quantity” allows ROI calculations (!)

Use these free ROI calculators…
http://www.humanfactors.com/downloads/roi.asp

ROI Calculator — Increased Productivity…
http://www.humanfactors.com/training/roi_incr_prodvt.htm

2. Remember:“Margin of Error”
Like voting polls… measures are fuzzy. Remember +/- 3%? (You need 1067 people needed to get +/-3%…)
If 8 of 10 test participants succeed, DON’T say “80% passed.”
Best you can say: 48% to 96% passed.
www.measuringusability.com/wald.htm

3. Use “Time-on-Task” for ROI
Reduces margin-of-error for ROI calcs.
Use “geometric mean” not “average” (just like “median house price”)
What’s your real target for time savings? (test)
http://www.measuringusability.com/time_intervals.php

4. Go for the “lower bound”…
www.humanfactors.com/downloads/quantitative-usability
Finally, success…

Converting UT Results to Business Measures
Efficiency (time-on-task) (DONE!)
Effectiveness (task success rates)
Errors (ouch)

1. Effectiveness (“Pass” Rate)
No real difference….

Need spectacular contrasts
http://www.measuringusability.com/comp_sample.php

Because these are — ”samples”
No real difference….

Better: compare with “population”
A real difference….

What’s the best we can say?
Using what criteria? 80%? 90%….

If high stakes: U need larger “N”
www.humanfactors.com/download/quantitative-usability
Your benchmark: 85% will succeed on task…

ROI = “profit only if usable by at least 85% of people who try…”

Effectiveness (task success rates) (DONE)
Errors (ouch)

3. Satisfaction (>80% = goal)
http://measuringuserexperience.com/
116/129 => 90 th Percentile scores of 81.2+ = 90 th Percentile

3. Satisfaction (goal: >80%)
http://en.wikipedia.org/wiki/System_Usability_Scale

3. Satisfaction (>80% = goal)
http://www.upassoc.org/usability_resources/conference/2004/UPA-2004-TullisStetson.pdf

Final Report: Satisfaction
The test of the High Point Quote application with 18 participants (and 17 results on the SUS) resulted in an average score of 79.7.
This compares very well with the 81.2 score that represents the 90 th percentile. We conclude that the application created a very good feeling of “usability” in our test participants.

Final Report: Success Rate
All 18 participants successfully completing the main task.
This implies that we can have 97.32% confidence of meeting our target that 85% of people like our participants will succeed.
We can also say that we have a 95% chance that at least 88% can do the task.
http://www.measuringusability.com/onep.php

Effectiveness (task success rates) (DONE)
Satisfaction (subjective ratings) (DONE)
Learning (time to learn) (See time-on-task)
Errors (See time-on-task)

UT Results in Business Terms
What percent of visitors can do the task?
Our income depends on your prediction!
Use estimates like….
95% confidence that 88% of end-users can do it!
60% chance that 85% can do it (uhmmmm)
95% chance that 60% of end-users can do it…
! Goforit…
E-mail your ideas: john@userexperienceworks.com

Getting Roi Numbers From Ut Des Moines10 Sept09

by johnsorflaten

Accessibility

Upload Details

Usage Rights

Statistics

Getting Roi Numbers From Ut Des Moines10 Sept09 Presentation Transcript