Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Usability Testing Fundamentals


Published on


Published in: Technology, Design
  • Be the first to comment

Usability Testing Fundamentals

  1. 1. Usability Testing FundamentalsEverything You Need to Know to Get Started Debbie Cook and Michelle Erickson 1
  2. 2. Introductions• Who we are: – Debbie Cook – Michelle Erickson• Who you are: – Name – Company – Role – Goals for this course 2
  3. 3. ObjectivesIn this workshop, you will learn to:• Plan, design, and prepare for a diagnostic usability test – Decide which user(s) to recruit – Create effective tasks and scenarios• Run and effectively moderate a usability test• Employ a low-cost, low-effort approach for capturing and analyzing results 3
  4. 4. Why Test? Kegworth Air Disaster - 1989 4
  5. 5. Why Test? Previous gauge design Updated gauge design 5
  6. 6. Agenda• Definition of usability testing• Planning, designing, and preparing for a test• BREAK• Moderating and running a test• Analyzing and summarizing results
  7. 7. What is Usability Testing? “A systematic way of observing actual and potential users of a product as they work with it under controlled conditions.”Observer Participant - Joe Dumas & Beth Loring Moderating Usability Tests Moderator
  8. 8. Two Types of Usability Tests • Formative test – Goal: Uncover and diagnose issues; understand user’s workflow and/or mental model – Focus is on qualitative assessment, identifying and fixing issues with the design • Summative test – Goal: Measure usability of a product or provide a baseline for comparison between products or versions – Focus is on quantitative assessmentThis tutorial will focus on Formative Testing. 8
  9. 9. Formative Testing Can Help You…• Learn about users’ requirements• Diagnose problems early and reduce waste• Ensure users’ success and improve their satisfaction with the product• Advocate for a change to the design• Educate stakeholders about how users interact with the product
  10. 10. Planning Moderating AnalyzingPLANNING, DESIGNING ANDPREPARING FOR YOUR TEST 10
  11. 11. Planning Moderating AnalyzingPlanning Roadmap Create a plan: Include the right “the what” people: “the who” • Objectives • Participants • Scenario • Observers • Tasks • Protocol • Prototype • Pilot test 11
  12. 12. Planning Moderating AnalyzingTest Plan Template 12
  13. 13. Planning Moderating AnalyzingCase Study:
  14. 14. Planning Moderating AnalyzingDeciding on Test Objectives• What are your goals for the test?• What do you hope to learn? – Are there aspects of the design you’re concerned about? – Are there particular user groups that you’re concerned about? 14
  15. 15. Planning Moderating AnalyzingThinking About Your UsersIdentify target user profiles of interest• Think about the target users of the product – Primary users – Secondary users• Determine which user characteristics are most likely to impact results• Review with team to ensure buy-in 15
  16. 16. Planning Moderating ActivityBrainstorm Objectives and UsersAs a group, let’s brainstorm together• What goals or objectives might you have for a usability test of this Web site?• Who are target users of this Web site? 16
  17. 17. Planning Moderating ExampleTest Objectives 17
  18. 18. Planning Moderating ExampleUsers 18
  19. 19. Planning Moderating AnalyzingCreating a Scenario• What is a test scenario? – A short description of the participant’s “current state” used to set the context for the test or task.• A well-written scenario… – Supports the goal(s) of your test – Is realistic – Speaks the user’s language, not the system’s 19
  20. 20. Planning Moderating AnalyzingDeveloping Your Tasks• What is a test task? – A brief description of what you want the participant to do during the test (not how to do it)• A well-written task… – Supports the goals of your test – Is realistic; it reflects a real task user will perform – Provides relevant details needed – Speaks the user’s language, not the system’s – Probes potential usability issues – Is typically independent of other test tasks 20
  21. 21. Planning Moderating AnalyzingTips for Developing TasksThink about tasks that are:• New to the user or require the user to change the way s/he performs the task• Performed frequently• Critical to success• Likely to reveal a potential usability issue or concern• Unique to a particular environment or interaction 21
  22. 22. Planning Moderating ActivityBrainstorm Scenario and TasksAs a group,• Brainstorm (15 minutes): – Test scenario - how would you “set the stage” for the user at the beginning of the test? – Tasks - what task(s) might you have the users perform to support your test objectives?• Report out (5 minutes): 22
  23. 23. Planning Moderating ExampleScenario and Tasks 23
  24. 24. Planning Moderating AnalyzingRefining Your TasksRefine and scope tasks based on:• Priority• Relevance• Prototype capabilities• Time – Estimated time to complete – Stakeholder availability• Possibility of completion NEVER give users an impossible task! 24
  25. 25. Planning Moderating AnalyzingEstablishing a Test Protocol• Always start the user off with an easy task• Identify possible curve balls – Create a plan for how you will deal with these if the user encounters them – Ensure the team and observers are aware of, and agree to, the protocol• Decide how you will provide help if asked• Identify key information for the moderator (you) and the observers to be aware of 25
  26. 26. Planning Moderating Analyzing Identifying Key Information For each task in your plan, capture: • The goal(s) for the task • The “correct” answer • Estimated time on task • Any special protocol for the task • Areas of interest you want your observers to take note ofThis information is only for you and your observers. 26
  27. 27. Planning Moderating AnalyzingExample Test Protocol 27
  28. 28. Planning Moderating ExampleKey Task InformationShow me how you would get started planning your trip.Max time on task: 10 minutesTask goals:• Learn how the user navigates to the site• Learn about how the user begins his/her search for informationProtocol Notes:• Start participant out with a clear desktop, i.e., no windows open, not even a browser.• This is an exploratory task to see how they get started, once the user starts down a particular path the facilitator will guide him/her to an appropriate follow up task.Observers’ take note of:• How does the user get to the Disney site? (e.g., Google search, typing in direct URL, etc.)• How does the user navigate to the site?• What does the user click on once s/he arrives at the site?• What information does s/he pay attention to first? What information does s/he start looking for?• Does the user browse for information or search?• Does the user notice the “Vacation Planning” option on the home page 28
  29. 29. Planning Moderating AnalyzingChoosing Participants• Identify user profiles of greatest interest• Determine which user characteristics are most likely to impact results• Review criteria with team to ensure buy-in Observer Participant Moderator 29
  30. 30. Planning Moderating AnalyzingDeciding on Number of Participants• It depends on… – Time – Money – Availability of participants that fit the profile – Attitude and availability of key stakeholders• Generally accepted practice = 6 – 12 participants – Three (3) to five (5) participants per profile – 80% of problems identified with 4 – 5 participants 30
  31. 31. Planning Moderating Analyzing Inviting Observers Invite people to observe and take notes • Team members – Developers – Designers Observer – Writers Participant – Quality Engineers – Product managers Moderator • Key stakeholders – Clients – ChampionsInvite anyone who might benefit from attending. 32
  32. 32. Planning Moderating ActivityBrainstorm Participants and ObserversIn a group,• Brainstorm (10 minutes): – Participants • Which user group(s) will the test focus on? • Who will you target as possible participants? – Observers – which stakeholders might you invite to observe the tests?• Report out (5 minutes): 33
  33. 33. Planning Moderating ExampleParticipants and Observers The project team 34
  34. 34. Planning Moderating Analyzing Planning RoadmapCreate a plan: Include the right “the what” people: “the who”• Objectives • Participants• Scenario • Observers• Tasks• Protocol• Prototype• Pilot test 35
  35. 35. BREAK 39
  36. 36. Planning Moderating AnalyzingMODERATING AND RUNNING A TEST 40
  37. 37. Planning Moderating AnalyzingModerator Responsibilities• Setting up the environment• Preparing observers• Caring for and briefing participants• Conducting the test Observer Participant Moderator 41
  38. 38. Planning Moderating AnalyzingSetting Up the Environment• Prototype • Paper prototype • Computer• Cameras• Audio and video recordersMyth: You need a state of the art usability lab torun tests.Reality: You can test a paper prototype anywhere! 42
  39. 39. Planning Moderating AnalyzingModerator Responsibilities• Setting up the environment• Preparing observers• Caring for and briefing participants• Conducting the test Observer Participant Moderator 43
  40. 40. Planning Moderating AnalyzingBriefing Observers• Set ground rules for etiquette• Teach observers how to take notes• Establish a protocol for observers’ questions during or after study • Have observers write notes on post-its and pass to moderator, or Instant Message during test • Allow observers to ask questions directly 44
  41. 41. Planning Moderating AnalyzingObserver Groundrules 45
  42. 42. Planning Moderating AnalyzingCollecting Observational DataTraining observers on taking objective notes • Indicate direct quotes vs. observation • Indicate if it is an observer thought or idea • Record task and context • Indicate time on tasks, when relevant • One observation per post-it 46
  43. 43. Planning Moderating AnalyzingWhat’s a Good Observation?Anything that you think is interesting,remarkable, noteworthy.• What you see the participant doing• What you hear the participant saying• What you notice the system doing in response to the participant’s actions 47
  44. 44. Planning Moderating AnalyzingExample Observation Notes 48
  45. 45. Planning Moderating AnalyzingModerator Responsibilities• Setting up the environment• Preparing observers• Briefing participants• Conducting the test Observer Participant Moderator 49
  46. 46. Planning Moderating AnalyzingBriefing Participants 50
  47. 47. Planning Moderating AnalyzingOverview• Caring for the participant• Informed consent and non-disclosure agreements• Using a test script• Introducing the test 51
  48. 48. Planning Moderating AnalyzingCaring for Participants• Reduce anxiety and create comfortable environment• Offer beverages, restrooms, and ensure their physical comfort• Answer questions when possible 52
  49. 49. Planning Moderating AnalyzingInformed Consent and Non-disclosureInform participants about:• Informed consent – Participant rights – Concept of minimal risk• Nondisclosure and confidentiality agreements Sample available on STC Website: Usability Toolkit: 53
  50. 50. Planning Moderating AnalyzingExample 54
  51. 51. Planning Moderating AnalyzingBenefits of Using a Test Script• Each participant hears all critical briefing information• All participants get the same information 55
  52. 52. Planning Moderating AnalyzingIntroducing the Test• Importance of their participation• Expected length of test• Test environment• Procedure (tasks, questions for participant, wrap up)• Breaks, stopping• Questions for the team 57
  53. 53. Planning Moderating AnalyzingThink Aloud Protocol• Participants verbalize thoughts and expectations• Moderator reminds participants and prompts for their thoughts 58
  54. 54. Planning Moderating AnalyzingWe’re Testing the Product!• Assure the participant that you’re testing the product, not their performance.• Most participants blame themselves when they encounter problems. 59
  55. 55. Planning Moderating ActivityParticipant Briefing Demo 60
  56. 56. Planning Moderating AnalyzingModerator Responsibilities• Setting up the environment• Preparing observers• Briefing participants• Conducting the test Observer Participant Moderator 61
  57. 57. Planning Moderating AnalyzingConducting the Test 62
  58. 58. Planning Moderating AnalyzingOverview• Task handling• Interacting with participants• Providing help• Concluding the test• Troubleshooting 63
  59. 59. Planning Moderating AnalyzingTask HandlingPace• Give tasks one at a time• Let participants set pace• Ask clarifying questions after each task (if needed) 65
  60. 60. Planning Moderating AnalyzingTask HandlingAssists and AbandonsBased on test goals, consider:• How will you provide assistance when the participant is stuck?• When will you advise or allow the participant to abandon a task or test?In general, let participant struggle.• Don’t assist if it interferes with an important objective. 66
  61. 61. Planning Moderating AnalyzingTask HandlingTask Completion• Participant should indicate when they are done with each task. • Ask them whether they are finished, if necessary. • Build confirmation into tasks.• If they do not know, moderator needs to determine whether the task is complete.• If not, ask them to reread and/or redo task. 67
  62. 62. Planning Moderating AnalyzingInteracting with ParticipantsHandling Questions• Asking unbiased questions: – Neutral words – Avoid adjectives and adverbs – Use both extremes (i.e. easy and difficult)• Answering participant questions: – Don’t answer directly – Turn questions around and ask the participant: What do you think will happen? What do you expect should happen? 69
  63. 63. Planning Moderating AnalyzingInteracting with ParticipantsInterruptingIn general, avoid interrupting the participant.But if you must:• Remind them to think aloud• Prompt them to clarify or reread tasks• Probe for information about their actions or expectations (“Say more about that…”) 70
  64. 64. Planning Moderating AnalyzingInteracting with ParticipantsUsing Active Listening Skills• Use nonverbal communication and minimal encouragers• Allow for silence, struggle, and uncertainty• Check expectations• Validate your understanding 71
  65. 65. Planning Moderating AnalyzingProviding Product HelpProvide minimal help• Only when and if participant requests helpUnless you are testing Help… 72
  66. 66. Planning Moderating AnalyzingConcluding the Test• Clarifying questions from you or observers• Post test interviews• Wrap up and closure 73
  67. 67. Planning Moderating ActivityModeration Demo 75
  68. 68. Planning Moderating AnalyzingDuring a Real Test…• Observers take notes on the content of the test, not the moderator’s performance.• Moderator’s primary responsibility is to facilitate the test and ensure the comfort of the participant – not take notes. 76
  69. 69. Planning Moderating ActivityModeration Demo and Debrief• Task 1: Observe moderator’s techniques – Use the checklist to takes notes.• Task 2: Capture observational data on post-its – What you see the participant doing – What you hear the participant saying – What you notice the system doing in response to the participant’s actions 77
  70. 70. Moderation Demo 78
  71. 71. Planning Moderating AnalyzingModeration Summary• You don’t need a fancy usability lab to test!• Brief observers on etiquette and note taking.• Make participants feel comfortable, inform them of their rights, and assure them you are testing the product, not them!• Let participants set the pace.• Interrupt only minimally and be careful not to bias the test! 79
  72. 72. Planning Moderating AnalyzingANALYZING AND SUMMARIZINGRESULTS 80
  73. 73. Planning Moderating AnalyzingGoals of Analysis and Summary• Identify the problems that were uncovered in the study• Develop a common understanding• Impact and improve design• Convince team to make changes 81
  74. 74. Planning Moderating AnalyzingAnalyzing Test DataAnalysis techniques vary depending on manyfactors:• Culture/organization• Test objectives and requirements• Your test goals 82
  75. 75. Planning Moderating AnalyzingOur Recommendation:Affinity Diagramming• Share and consolidate observation notes.• Categorize notes into meaningful groups.• Prioritize by voting on top issues.• Record and communicate results to team and stakeholders. 83
  76. 76. Planning Moderating ActivityAffinitize the Results• Post all notes together, uncategorized. Toss duplicates.• Categorize notes into meaningful groups. – When you see 2 that are related, take both notes and post together. – As soon as you put two notes together, title the category on a 3” x 3” post-it note.• Prioritize the feedback (by category or individual note). 84
  77. 77. Planning Moderating AnalyzingReporting ResultsReporting techniques vary depending on time,culture, and resources.Some examples include: • Video clips • Formal detailed report • Excel spreadsheet • Executive summary • PowerPoint presentation 85
  78. 78. Planning Moderating AnalyzingAnalysis and Reporting Summary• Goals of analysis: – Develop a common understanding – Identify the problems – Convince team to make changes – Improve a design• Scale reports to fit.• Try Affinity diagramming! 86
  79. 79. CONCLUSION 87
  80. 80. Just Test!• Show your product to users and get feedback.• Use these materials to refine your testing methods.• You don’t need to be an expert.• You don’t need a state-of-the-art lab.• You don’t need elaborate protocols and test scripts.• The more you test, the better you’ll get. 88
  81. 81. Primary ResourcesA Practical Guide to Moderating Usability TestsUsability Testing Joseph S. DumasJoseph S. Dumas & Beth Loring& Janice C. Redish 89
  82. 82. Thank you!Contact us with any questions…•• 95