„Five users will find 85% of the usability problems” - and other myths about usability testing

  • 1,290 views
Uploaded on

Prezentacja Rolfa Molicha z DialogDesign w ramach World Usability Day 2011 we Wrocławiu

Prezentacja Rolfa Molicha z DialogDesign w ramach World Usability Day 2011 we Wrocławiu

More in: Technology , Design
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,290
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
0
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. World Usability Day Wroclaw 2011”Five users will find 85% ofthe usability problems” - Rolf Molichand other myths about DialogDesignusability testing
  • 2. Dogma 1► Usability professionals must be highly usable Reports, presentations, responses to comments, etc. 2
  • 3. Źdźbło w okubliźniego widzisz, a belki w swoim nie dostrzegasz Sweep before your own door before you sweep the doorsteps of your neighbors
  • 4. When you point your finger at someone, remember that three fingers point at yourself
  • 5. Dogma 2► Data trumps opinion at any time 5
  • 6. A Good, Ol’ American Bake-off
  • 7. A Usability Bake-off
  • 8. CUE-9:19 Atlanta +16 Chemnitz Professionals 5 Videos>100 Different Findings
  • 9. Data to Fight UnsubstantiatedMyths and Voodoo -- CUE motto
  • 10. Excerpt from Usability Test Report - CUE-6, Team C
  • 11. Excerpt from Usability Test Report - CUE-6, Team G
  • 12. Excerpt from Usability Test Report - CUE-6, Team G
  • 13. Five users are enough to catch 1 85% of the usability problems in practically any product Agree Both agree & disagree DisagreeTruth or myth?
  • 14. Five users are enough to catch 1 85% of the usability problems in practically any product Agree Both agree & disagree DisagreeTruth or myth?
  • 15. ”The parabola of optimism” (Jared Spool)
  • 16. Real data: CUE studiesCUE = Comparative Usability Evaluation► 10-17 professional teams test the same website.► In all CUE-studies, more than 60% of problems were uniquely reported.► In CUE-4, more than 300 issues were reported. 16
  • 17. The main goal of a usability test is 2 to discover usability problems Agree Both agree & disagree DisagreeTruth or myth?
  • 18. The main goal of a usability test is 2 to discover usability problems Agree Both agree & disagree DisagreeTruth or myth?
  • 19. Real cowboyprogrammers don’t need no stinkin’ usability test -- Anonymous
  • 20. Goals of usability test1. Find problems so they can be corrected2. Demonstrate convincingly to skeptical stakeholders (developers, management) that problems exist and that there are efficient ways to find and correct them 20
  • 21. Expert reviews provide results that 3 are as reliable as those from usability tests Agree Both agree & disagree DisagreeTruth or myth?
  • 22. Expert reviews provide results that 3 are as reliable as those from usability tests Agree Both agree & disagree DisagreeTruth or myth?
  • 23. Expert review vs. UtestData from CUE-studies, most notably CUE-4,consistently shows► Expert reviews find as many problems as usability testing► Expert reviews overlook problems – but so do usability tests► Expert reviews are cheaper than usability testsThese findings apply only when expert reviewsare carried out by Experts 23
  • 24. Eye-tracking shows us what users 4 see, so it reveals important issues that usability testing without it cant Agree Both agree & disagree DisagreeTruth or myth?
  • 25. Eye-tracking shows us what users 4 see, so it reveals important issues that usability testing without it cant Agree Both agree & disagree DisagreeTruth or myth?
  • 26. Real data: Eye tracking► CUE-6 tested Enterprise.com, a large American car rental website► One team (team G) used eye tracking while the other 12 teams used conventional evaluation methods► Team G was from a renowned promoter of eye tracking► Team G did not report any important findings that were not reported by other teams 26
  • 27. At least 25% of the comments in a 5 usability test report should be positive Agree Both agree & disagree DisagreeTruth or myth?
  • 28. At least 25% of the comments in a 5 usability test report should be positive Agree Both agree & disagree DisagreeTruth or myth?
  • 29. Positive Findings1. Make it easier for you to sell inconvenient truths2. Ensure that the development team doesn’t remove a feature that users actually liked 29
  • 30. Positive Findings „Łyżka cukru pozwoli przełknąć ci lekarstwo” ”A spoonful of sugar helps the medicine go down” Mary Poppins 30
  • 31. Even developershave feelings
  • 32. Required by UPA CPCPositive findings are required by theUPA Code of Professional Conduct► The CPC requires you to "accurately report both the positive and negative feedback".► If you can prove that there is nothing positive to say, then you have fulfilled the CPC. I am not sure how I would prove this. 32
  • 33. When you have a hammer in your hand, everything looks like a nail -- The Law of the Instrument, Abraham Kaplan
  • 34. Real World DataNumber of reported positive findings for each CUE-9 team 50% 30% 10%0 0 1 1 2 2 2 2 2 3 3 4 4 5 8 10 34
  • 35. Usability testing can be conducted 6 by anyone Agree Both agree & disagree DisagreeTruth or myth?
  • 36. Usability testing can be conducted 6 by anyone Agree Both agree & disagree DisagreeTruth or myth?
  • 37. Quality testing can be conducted 6 by anyone Agree Both agree & disagree DisagreeTruth or myth?
  • 38. If you don’t care about quality,everything else is trivial -- Gerald M. Weinberg
  • 39. 10 Assessments1 professional passed with flying colors2 professionals had serious deviations but no critical deviations from generally accepted good practice in usability testing.5 professionals had both critial deviations and serious deviations2 professionals did so badly that the assessor recommended that the client should not pay for the service, or that the test should be repeated at no cost for the client.A "serious deviation" from good practice may skew important results.A "critical deviation" from good practice will skewimportant test results seriously. 39
  • 40. Client ActionsActions taken by clients to prevent future quality problems:► Develop and publish usable guidelines for good usability testing► Quality Assurance as part of procurement “Do a test while we observe” Is their understanding of a good usability test the same as ours?► Make assessments available to teams and insist that they respond.Fortunately, most usability professionals take pride in doing things right 40
  • 41. You can’tcontrol what you don’t measure -- Tom DeMarco
  • 42. UTA ChecklistSeven out of currently about 70 checkpoints:► Do tasks contain “humorous” elements?► Do tasks contain hidden clues?► Is the moderator sufficiently curious?► Does the moderator provide hidden clues?► Does the moderator talk too much?► If both a moderator and a note taker attend the test, are their notes consolidated after the test?► Can all findings in the test report be traced back to actual events during the test? 42
  • 43. Certification► UPA-I is currently working with the German UPA on a professional certification scheme► Our friends at Human Factors International (HFI) have proven that there is some interest in professional certification in the market, for example from employers 43
  • 44. A burning question► Are there sufficiently many interesting things that we agree on in our field to make certification meaningful? 44
  • 45. “My usability testing skills are7 above average” Agree Both agree & disagree DisagreeEvaluation
  • 46. Dziękuję.Życzę miłego dnia!
  • 47. Did this presentation change some8 of your views of usability testing? Yes Don’t know NoEvaluation
  • 48. Takeaways► As usability professionals we must set a good example for the world► Our products, presentations, doings must be usable and useful: Correct, relevant, brief, constructive► Five users are enough to drive a useful iterative cycle. Never claim that you can find all usability problems► Expert reviews are valuable – but politically challenging► Remember the positive findings  48