World Usability Day Wroclaw 2011”Five users will find 85% ofthe usability problems” -          Rolf Molichand other myths ...
Dogma 1► Usability professionals must be highly usable   Reports, presentations, responses to comments, etc.              ...
Źdźbło w okubliźniego widzisz, a belki w swoim nie dostrzegasz   Sweep before your own door        before you sweep the do...
When you point your finger at someone, remember that three fingers point at yourself
Dogma 2► Data trumps opinion at any time                                    5
A Good, Ol’ American Bake-off
A Usability Bake-off
CUE-9:19 Atlanta +16 Chemnitz Professionals    5 Videos>100 Different  Findings
Data to Fight UnsubstantiatedMyths and Voodoo           -- CUE motto
Excerpt from Usability Test Report - CUE-6, Team C
Excerpt from Usability Test Report - CUE-6, Team G
Excerpt from Usability Test Report - CUE-6, Team G
Five users are enough to catch 1                 85% of the usability problems in                 practically any product ...
Five users are enough to catch 1                 85% of the usability problems in                 practically any product ...
”The parabola of optimism” (Jared Spool)
Real data: CUE studiesCUE = Comparative Usability Evaluation► 10-17 professional teams test the same website.► In all CUE-...
The main goal of a usability test is 2                 to discover usability problems                     Agree           ...
The main goal of a usability test is 2                 to discover usability problems                     Agree           ...
Real cowboyprogrammers don’t need no stinkin’   usability test           -- Anonymous
Goals of usability test1.   Find problems so they can be corrected2.   Demonstrate convincingly to skeptical     stakehold...
Expert reviews provide results that 3                 are as reliable as those from                 usability tests       ...
Expert reviews provide results that 3                 are as reliable as those from                 usability tests       ...
Expert review vs. UtestData from CUE-studies, most notably CUE-4,consistently shows► Expert reviews find as many problems ...
Eye-tracking shows us what users 4                 see, so it reveals important issues                 that usability test...
Eye-tracking shows us what users 4                 see, so it reveals important issues                 that usability test...
Real data: Eye tracking► CUE-6 tested Enterprise.com, a large  American car rental website► One team (team G) used eye tra...
At least 25% of the comments in a 5                 usability test report should be                 positive              ...
At least 25% of the comments in a 5                 usability test report should be                 positive              ...
Positive Findings1.   Make it easier for you to sell inconvenient truths2.   Ensure that the development team doesn’t     ...
Positive Findings „Łyżka cukru pozwoli przełknąć ci lekarstwo”      ”A spoonful of sugar helps the medicine go down”      ...
Even developershave feelings
Required by UPA CPCPositive findings are required by theUPA Code of Professional Conduct► The CPC requires you to "accurat...
When you have a hammer in your hand, everything looks like a nail                              -- The Law of the Instrumen...
Real World DataNumber of reported positive findings for each CUE-9 team                                                   ...
Usability testing can be conducted 6                 by anyone                    Agree                    Both agree & di...
Usability testing can be conducted 6                 by anyone                    Agree                    Both agree & di...
Quality testing can be conducted 6                 by anyone                    Agree                    Both agree & disa...
If you  don’t care about quality,everything else   is trivial      -- Gerald M. Weinberg
10 Assessments1 professional passed with flying colors2 professionals had serious deviations but no critical deviations fr...
Client ActionsActions taken by clients to prevent future quality problems:► Develop and publish usable guidelines for good...
You can’tcontrol what you don’t measure      -- Tom DeMarco
UTA ChecklistSeven out of currently about 70 checkpoints:► Do tasks contain “humorous” elements?► Do tasks contain hidden ...
Certification► UPA-I is currently working with the German  UPA on a professional certification scheme► Our friends at Huma...
A burning question► Are there sufficiently many interesting things  that we agree on in our field to make  certification m...
“My usability testing skills are7             above average”                 Agree                 Both agree & disagree  ...
Dziękuję.Życzę miłego   dnia!
Did this presentation change some8             of your views of usability testing?                 Yes                 Don...
Takeaways► As usability professionals we must set a good example for  the world► Our products, presentations, doings must ...
Upcoming SlideShare
Loading in …5
×

„Five users will find 85% of the usability problems” - and other myths about usability testing

1,802 views

Published on

Prezentacja Rolfa Molicha z DialogDesign w ramach World Usability Day 2011 we Wrocławiu

Published in: Technology, Design
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,802
On SlideShare
0
From Embeds
0
Number of Embeds
337
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

„Five users will find 85% of the usability problems” - and other myths about usability testing

  1. 1. World Usability Day Wroclaw 2011”Five users will find 85% ofthe usability problems” - Rolf Molichand other myths about DialogDesignusability testing
  2. 2. Dogma 1► Usability professionals must be highly usable Reports, presentations, responses to comments, etc. 2
  3. 3. Źdźbło w okubliźniego widzisz, a belki w swoim nie dostrzegasz Sweep before your own door before you sweep the doorsteps of your neighbors
  4. 4. When you point your finger at someone, remember that three fingers point at yourself
  5. 5. Dogma 2► Data trumps opinion at any time 5
  6. 6. A Good, Ol’ American Bake-off
  7. 7. A Usability Bake-off
  8. 8. CUE-9:19 Atlanta +16 Chemnitz Professionals 5 Videos>100 Different Findings
  9. 9. Data to Fight UnsubstantiatedMyths and Voodoo -- CUE motto
  10. 10. Excerpt from Usability Test Report - CUE-6, Team C
  11. 11. Excerpt from Usability Test Report - CUE-6, Team G
  12. 12. Excerpt from Usability Test Report - CUE-6, Team G
  13. 13. Five users are enough to catch 1 85% of the usability problems in practically any product Agree Both agree & disagree DisagreeTruth or myth?
  14. 14. Five users are enough to catch 1 85% of the usability problems in practically any product Agree Both agree & disagree DisagreeTruth or myth?
  15. 15. ”The parabola of optimism” (Jared Spool)
  16. 16. Real data: CUE studiesCUE = Comparative Usability Evaluation► 10-17 professional teams test the same website.► In all CUE-studies, more than 60% of problems were uniquely reported.► In CUE-4, more than 300 issues were reported. 16
  17. 17. The main goal of a usability test is 2 to discover usability problems Agree Both agree & disagree DisagreeTruth or myth?
  18. 18. The main goal of a usability test is 2 to discover usability problems Agree Both agree & disagree DisagreeTruth or myth?
  19. 19. Real cowboyprogrammers don’t need no stinkin’ usability test -- Anonymous
  20. 20. Goals of usability test1. Find problems so they can be corrected2. Demonstrate convincingly to skeptical stakeholders (developers, management) that problems exist and that there are efficient ways to find and correct them 20
  21. 21. Expert reviews provide results that 3 are as reliable as those from usability tests Agree Both agree & disagree DisagreeTruth or myth?
  22. 22. Expert reviews provide results that 3 are as reliable as those from usability tests Agree Both agree & disagree DisagreeTruth or myth?
  23. 23. Expert review vs. UtestData from CUE-studies, most notably CUE-4,consistently shows► Expert reviews find as many problems as usability testing► Expert reviews overlook problems – but so do usability tests► Expert reviews are cheaper than usability testsThese findings apply only when expert reviewsare carried out by Experts 23
  24. 24. Eye-tracking shows us what users 4 see, so it reveals important issues that usability testing without it cant Agree Both agree & disagree DisagreeTruth or myth?
  25. 25. Eye-tracking shows us what users 4 see, so it reveals important issues that usability testing without it cant Agree Both agree & disagree DisagreeTruth or myth?
  26. 26. Real data: Eye tracking► CUE-6 tested Enterprise.com, a large American car rental website► One team (team G) used eye tracking while the other 12 teams used conventional evaluation methods► Team G was from a renowned promoter of eye tracking► Team G did not report any important findings that were not reported by other teams 26
  27. 27. At least 25% of the comments in a 5 usability test report should be positive Agree Both agree & disagree DisagreeTruth or myth?
  28. 28. At least 25% of the comments in a 5 usability test report should be positive Agree Both agree & disagree DisagreeTruth or myth?
  29. 29. Positive Findings1. Make it easier for you to sell inconvenient truths2. Ensure that the development team doesn’t remove a feature that users actually liked 29
  30. 30. Positive Findings „Łyżka cukru pozwoli przełknąć ci lekarstwo” ”A spoonful of sugar helps the medicine go down” Mary Poppins 30
  31. 31. Even developershave feelings
  32. 32. Required by UPA CPCPositive findings are required by theUPA Code of Professional Conduct► The CPC requires you to "accurately report both the positive and negative feedback".► If you can prove that there is nothing positive to say, then you have fulfilled the CPC. I am not sure how I would prove this. 32
  33. 33. When you have a hammer in your hand, everything looks like a nail -- The Law of the Instrument, Abraham Kaplan
  34. 34. Real World DataNumber of reported positive findings for each CUE-9 team 50% 30% 10%0 0 1 1 2 2 2 2 2 3 3 4 4 5 8 10 34
  35. 35. Usability testing can be conducted 6 by anyone Agree Both agree & disagree DisagreeTruth or myth?
  36. 36. Usability testing can be conducted 6 by anyone Agree Both agree & disagree DisagreeTruth or myth?
  37. 37. Quality testing can be conducted 6 by anyone Agree Both agree & disagree DisagreeTruth or myth?
  38. 38. If you don’t care about quality,everything else is trivial -- Gerald M. Weinberg
  39. 39. 10 Assessments1 professional passed with flying colors2 professionals had serious deviations but no critical deviations from generally accepted good practice in usability testing.5 professionals had both critial deviations and serious deviations2 professionals did so badly that the assessor recommended that the client should not pay for the service, or that the test should be repeated at no cost for the client.A "serious deviation" from good practice may skew important results.A "critical deviation" from good practice will skewimportant test results seriously. 39
  40. 40. Client ActionsActions taken by clients to prevent future quality problems:► Develop and publish usable guidelines for good usability testing► Quality Assurance as part of procurement “Do a test while we observe” Is their understanding of a good usability test the same as ours?► Make assessments available to teams and insist that they respond.Fortunately, most usability professionals take pride in doing things right 40
  41. 41. You can’tcontrol what you don’t measure -- Tom DeMarco
  42. 42. UTA ChecklistSeven out of currently about 70 checkpoints:► Do tasks contain “humorous” elements?► Do tasks contain hidden clues?► Is the moderator sufficiently curious?► Does the moderator provide hidden clues?► Does the moderator talk too much?► If both a moderator and a note taker attend the test, are their notes consolidated after the test?► Can all findings in the test report be traced back to actual events during the test? 42
  43. 43. Certification► UPA-I is currently working with the German UPA on a professional certification scheme► Our friends at Human Factors International (HFI) have proven that there is some interest in professional certification in the market, for example from employers 43
  44. 44. A burning question► Are there sufficiently many interesting things that we agree on in our field to make certification meaningful? 44
  45. 45. “My usability testing skills are7 above average” Agree Both agree & disagree DisagreeEvaluation
  46. 46. Dziękuję.Życzę miłego dnia!
  47. 47. Did this presentation change some8 of your views of usability testing? Yes Don’t know NoEvaluation
  48. 48. Takeaways► As usability professionals we must set a good example for the world► Our products, presentations, doings must be usable and useful: Correct, relevant, brief, constructive► Five users are enough to drive a useful iterative cycle. Never claim that you can find all usability problems► Expert reviews are valuable – but politically challenging► Remember the positive findings  48

×