Successfully reported this slideshow.
Your SlideShare is downloading. ×

Usability Testing On A Digital Product

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 179 Ad

Usability Testing On A Digital Product

Download to read offline

This was a 4-hour workshop that was given at World Usability Day Colombia. #wudco14

Summary:
Now more than ever is the survival of the easiest. Whether the product is a website or a handheld device, success depends largely on how easy it is to use. Usability testing is one of the most effective for creating an intuitive methods. By observing actual people when they use the product, you can get valuable insights if your design is easy to use. Attendees will learn how to conduct a usability test with end users of a product. This workshop is highly interactive and includes several practical exercises to give participants practical experience.

You will learn:
- How to plan a usability testing study
- How to define the goals and objectives
- Explore options (unmoderated usability testing vs. unmoderated & remote vs. in-person)
- How to recruit the right participants
- How to create tasks (Interview-based vs. predefined tasks)
- How to moderate a usability test
- How to analyze and report the results

This was a 4-hour workshop that was given at World Usability Day Colombia. #wudco14

Summary:
Now more than ever is the survival of the easiest. Whether the product is a website or a handheld device, success depends largely on how easy it is to use. Usability testing is one of the most effective for creating an intuitive methods. By observing actual people when they use the product, you can get valuable insights if your design is easy to use. Attendees will learn how to conduct a usability test with end users of a product. This workshop is highly interactive and includes several practical exercises to give participants practical experience.

You will learn:
- How to plan a usability testing study
- How to define the goals and objectives
- Explore options (unmoderated usability testing vs. unmoderated & remote vs. in-person)
- How to recruit the right participants
- How to create tasks (Interview-based vs. predefined tasks)
- How to moderate a usability test
- How to analyze and report the results

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Advertisement

Similar to Usability Testing On A Digital Product (20)

Advertisement

Usability Testing On A Digital Product

  1. 1. For a digital product usability testing World Usability day colombia 2014 Kyle Soucy @kylesoucy
  2. 2. Image source: facit digital What is Usability Testing?
  3. 3. How do you conduct a usability study?
  4. 4. Lets us begin our Journey...
  5. 5. Plan Study Define goals and objectives
  6. 6. What do you want to learn? How do teachers shop for books? Do they use these slider dots? Do they notice these arrows?
  7. 7. WhatYou Can Learn From A Usability Study
  8. 8. Do users understand how to complete a desired task?
  9. 9. Does the design prevent users from completing tasks more effectively?
  10. 10. Do users notice the buttons or links that you want them to click on?
  11. 11. Are users engaged in the content and understand it?
  12. 12. How long does it take for a user to complete a task?
  13. 13. Does the interface cause users undo frustration or confusion?
  14. 14. WhatYou Can’t Learn From A Usability Study
  15. 15. Opinions or other subjective data (We’re not testing hundreds or thousands of people)
  16. 16. The future (We can only report on what we directly observe)
  17. 17. Design preferences (You won’t get answers to design questions)
  18. 18. Why can’t I just use a focus group?
  19. 19. Qualitative vs. Quantitative Research • Qualitative Research - Data gathered directly by observing the user - Researcher can ask follow-up questions, probe on behavior - Analysis of data is not mathematical (Why? How? Compare...) • Quantitative Research - Data gathered indirectly through surveys, log files, support calls, etc. - Data can be averaged or statistically tested Source: Christian Rohrer Move, Inc. | Realor.com
  20. 20. Understanding The Domain And The Product
  21. 21. If you don’t know the product, you can’t test it!
  22. 22. Plan Study Define what you are testing
  23. 23. Low vs. High-Fidelity Prototypes
  24. 24. Plan Study Task design
  25. 25. 1. Write a list of the most important tasks that users need to accomplish using the product. 2. Turn each task into a scenario. a) Provide some context. b) Supply needed information. c) Don’t give any clues or hints! 3. Decide on the order of tasks. a) Make the first task easier.
  26. 26. Bad Task: Buy a book about genealogy online.
  27. 27. Good Task: You need to get a book about genealogy that covers the reliability of different sources.You want this book in your hands by Thursday. Find and purchase this book.
  28. 28. What makes a good task: • You’re asking users to do something that they would actually do in real life. • Good scope. Not too broad, not too specific. • Has a clear end point. • Elicits action, not opinion. Don’t start with “how would you” or “where would you”
  29. 29. Good Task: You need to get a book about genealogy that covers the reliability of different sources.You want this book in your hands by Thursday. Find and purchase this book.Find and purchase this book. ✓ Has a clear end point ✓ Elicits action, not opinion ✓ Good scope
  30. 30. Your Turn... 1. Take out a sheet of paper (or your computer, phone, or iPad) 2. Write down the most important task your users perform on your product. 3. Turn the task into a scenario. a) Provide some context. b) Supply needed information. c) Don’t give any clues or hints!
  31. 31. Other things to consider...
  32. 32. Restrictions: Consider whether or not you want to place restrictions on how participants try and accomplish the tasks (e.g.“don’t use search” or “stay on the website”).
  33. 33. Printing: If you’re conducting an in-person test, you may want to print out the scenarios so the participant can refer back to it.
  34. 34. Time-Fillers: Create more tasks than you think you have time to complete just incase someone finishes early. Every minute with a participant is an opportunity to learn something.
  35. 35. Visual Aids: If you’re worried that revealing the name of something will bias the results, consider showing the participant a picture instead.
  36. 36. Visual Aid Example: A friend of yours uses on of these to make baby food, so you’d like to buy one...
  37. 37. Alternating Tasks/Prototypes: Depending on what you’re testing, you may need to alternate tasks. Participants are learning the interface as they go, so the last task may always be the easiest to accomplish unless you alternate them.
  38. 38. Interview-based vs. Predefined Tasks
  39. 39. Interview-based Tasks 1. Interview participant about how they use the product. 2. Use the answers given in the interview to create tasks on the fly. When in doubt just ask,“what was the last thing you did when you used this product? Can you show me?”
  40. 40. Plan Study Define participants
  41. 41. Demographics
  42. 42. Cover all user groups - Prospective as well as existing users
  43. 43. Be specific and get buy-in from stakeholders!
  44. 44. Recruiting Options
  45. 45. Do ItYourself
  46. 46. Hire a recruiter Warning: It’s never this easy!
  47. 47. Live-intercept Study
  48. 48. How many to test?
  49. 49. Source: Jakob Nielsen
  50. 50. Budget Allocation What NOT to do... 1 Round of Testing
  51. 51. Budget Allocation What you SHOULD do... Round 1 Round 2 Round 3 Design Iteration Design Iteration
  52. 52. What to pay participants?
  53. 53. Plan Study define testing method
  54. 54. Vs. Remote vs. In-person Testing
  55. 55. Remote Testing Pros & Cons Participants stay in their native environment. Accessible to a larger & more diverse pool of participants. Recruiting is easier. Less no shows & easier last minute replacements. Opportunity to gather ethnographic data. Cost & time savings. No travel & lab rental fees. Easier for observers to “attend” a session. Risk of compromising data because the participant is using a foreign PC is removed. Pros: Can’t see the user’s facial expressions. Can’t see if the user is using peripheral devices. Dependent on an Internet connection. Can only recruit participants with a high- speed Internet connection. Cons:
  56. 56. Remote Testing Tools • GoToMeeting • WebEx • Adobe Connect • Join.me • Many more...
  57. 57. Remote Testing Tips • If using a cordless phone, have multiple handsets charged and ready • Use a phone recording controller • Use as few tools as possible • Back up, back up, back up! • USB Modem • Screen sharing backup
  58. 58. • Create a test meeting to make sure the participant can connect beforehand • Instruct participants to log on early and use a landline (noVOIP) • Turn off entry/exit chimes for conference call • Allow only one observer to chat to you • Use a separate chat/IM client
  59. 59. Unmoderated vs. Moderated Testing
  60. 60. What is Unmoderated Remote Testing?
  61. 61. Unmoderated Testing Tools (There are a lot!)
  62. 62. We Won’t Talk About...
  63. 63. We Will Talk About...
  64. 64. Has Anyone Here Used These Tools Before?
  65. 65. WhatYou Can Learn (Depends on the tool and how you use it!)
  66. 66. Web Analytics Image source: http://www.hiero.com/web-analytics.html
  67. 67. Task Completion Rates Image source: Loop11
  68. 68. Number of Clicks/ Clickstream Image source: Userlytics
  69. 69. Time on Task/Page Image source: http://blog.usabilla.com/underdogs-beat-expedia-in-usability-test/
  70. 70. Satisfaction Ratings/ Opinion Rankings Image source: http://wikidoc.org/index.php/Pain It is simple to use: Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree
  71. 71. True Intent Image source: http://ethnio.com/images/live_screener_example_page1.jpg
  72. 72. In-context Feedback Image source: Loop11 “...I’m only offered a search by model number. How ridiculous.” “...I would like to see the products availability and delivery/ shipping information listed.”
  73. 73. How Actionable is the Data? (Depends on the questions you ask!)
  74. 74. Conducting Unmoderated Remote Usability Tests
  75. 75. Process 1. Define the study 2. Recruit participants 3. Launch test and send email invites 4. Analyze results 3. Run a Pilot test! > 4 / 5 /
  76. 76. Benefits
  77. 77. Test hundreds of people simultaneously while keeping them in their own natural environment.
  78. 78. Test multiple websites simultaneously.
  79. 79. Reduced costs... depending on the tool you use.
  80. 80. Great way to plant the seed of UCD methodologies and introduce usability testing into a company. UX
  81. 81. Fewer logistics to manage and no need to worry about no-shows and getting last-minute replacements.
  82. 82. Fast Results
  83. 83. Drawbacks
  84. 84. Nothing beats watching participants in real time and being able to ask probing questions about what they are doing as it’s happening.
  85. 85. Some participants may only be interested in earning the honorarium you’ve provided as an incentive. $$$
  86. 86. You cannot conduct interview-based tasks.
  87. 87. It is simple to use: Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree Image source: http://www.hiero.com/web-analytics.html What participants report on surveys can be very different in comparison to what they actually do.
  88. 88. Image source: http://www.hiero.com/web-analytics.html It’s possible for participants to think they’ve successfully completed a task when they haven’t.
  89. 89. Image source: http://www.hiero.com/web-analytics.html Does it matter if it took someone longer to complete a task?
  90. 90. When to Conduct Automated Testing?
  91. 91. Not a replacement for moderated testing or research. Best used to compliment your qualitative research.
  92. 92. Unmoderated Testing Tools (A closer look...)
  93. 93. Automated Testing Tools with Recruiting Panels
  94. 94. The danger... I wonder how a real customer would actually use this website? Panelist
  95. 95. UserTesting.comVideo
  96. 96. BLOCKED BLOCKED The danger...
  97. 97. There are no shortcuts to conducting good research.
  98. 98. Automated Testing Tools with Other Recruiting Options
  99. 99. Loop11
  100. 100. Treejack
  101. 101. Chalkmark
  102. 102. Chalkmark Results
  103. 103. Automated Tools with Audio andVideo
  104. 104. Handout: Overview of Unmoderated Usability Testing Tools /usableinterface
  105. 105. Plan Study schedule participants
  106. 106. Example Testing Schedule
  107. 107. Plan Study invite participants and observers
  108. 108. Key to Success… Attendance of product decision makers during testing!
  109. 109. Example Remote Participant Invite
  110. 110. Set Up Study
  111. 111. Usability “Labs”
  112. 112. Image source: facit digital Source: Kent State University Vs.
  113. 113. Ideal “Lab-Less” Set Up
  114. 114. Should observers be in the room?
  115. 115. Video Recording Options • Techsmith Morae - $1,995 USD • Techsmith Camtasia - $99 USD • Silverback (Mac only) - v2 Free • Online Meeting Built-in Recorders
  116. 116. ExampleVideo Consent Form
  117. 117. Example Moderator Guide
  118. 118. Moderate Study
  119. 119. Greet Participant and Give Introduction
  120. 120. Preliminary Interview
  121. 121. Evaluation Instruction
  122. 122. Think Aloud Techniques
  123. 123. Think Aloud Techniques Concurrent (CTA) Retrospective (RTA) Vs.
  124. 124. Variation of RTA... Show the participant their video after they’re done testing.
  125. 125. Reminder Prompting • So...? • So, what are you thinking? • What are you seeing here? • What are you looking at? • Can you tell me what you think is happening here? Source: Moderating Usability Testing by Joe Dumas and Beth Loring
  126. 126. Probing Questions • Is that what you expected? • What would you do next? • What did you think about that task? • Help me to understand more about... Source: Moderating Usability Testing by Joe Dumas and Beth Loring
  127. 127. Encouraging Statements • This feedback is really helpful. • Thank you for thinking out loud. • That’s good to know. • You’re doing fine. Source: Moderating Usability Testing by Joe Dumas and Beth Loring
  128. 128. Remain Neutral • When in doubt turn the question around: • “I don’t know what do you think?” • “What would you do if I wasn’t here? • Keep acknowledgements limited to:“ah”, “Mm Hmm”,“Ok”, etc. Source: Moderating Usability Testing by Joe Dumas and Beth Loring and Rocket Surgery Made Easy by Steve Krug
  129. 129. Letting Participants Struggle
  130. 130. Note-taking
  131. 131. Analyze & Report Findings
  132. 132. Formal Usability Testing Report Findings categorized by screens or pages
  133. 133. When, What, Who, Where, ! and Why Statement! 3-4 ! Positive Findings! 3-4 ! Negative Findings!
  134. 134. Major Usability Problem
  135. 135. Video Clip Example
  136. 136. Mobile Usability Testing
  137. 137. Image source: http://img.gawkerassets.com/img/181rzskxifcohjpg/original.jpg
  138. 138. “...the mobile web will be bigger than desktop Internet use by 2015.” - Morgan Stanley Study Source: http://mashable.com/2010/04/13/mobile-web-stats/Source: http://mashable.com/2010/04/13/mobile-web-stats/
  139. 139. Image source: http://3.bp.blogspot.com/-Oh6rZE2fG1k/Uct4T4KhE0I/AAAAAAAAARo/3c_HFDkMwAE/s1600/girls-on-their-phone.jpg Mobile matters!
  140. 140. Mobile In-person Usability Testing Image source: http://unmatchedstyle.com/news/usability-testing-on-mobile-devices.php
  141. 141. Image source: http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/
  142. 142. Testing on paper prototypes Image source: http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/
  143. 143. What about remote mobile usability testing?
  144. 144. Current Navigation Tabbed Navigation Hidden Navigation
  145. 145. What about remote mobile usability testing? unmoderated >
  146. 146. Other Mobile Usability TestingTools
  147. 147. Image source: http://blog.larrybodine.com/uploads/image/a-99-year-old-lady-using-the-ipad.jpg
  148. 148. Fitting Research Into An Agile Process
  149. 149. Waterfall Analysis Design Coding Testing Analysis Design Coding Testing Agile Analysis Design Coding Testing Analysis Design Coding Testing 1-4 week sprints
  150. 150. Parallel Sprints #1
  151. 151. Sprint 1Dev: UX: Sprint 2 Sprint 3 Release Sprint 1 Sprint 2 Sprint 3
  152. 152. Staggered Sprints #2
  153. 153. Source: http://www.upassoc.org/upa_publications/jus/2007may/agile-ucd.pdf Tip: Choose a feature to develop first that has a lengthy dev sprint.
  154. 154. Testing On A Small Budget
  155. 155. • Conduct your own recruiting • Don’t offer large honorariums • Don’t use a lab • Don’t video record • Test less people • Use surrogate users or just conduct cognitive walkthroughs • Conduct unmoderated studies • Don’t write formal reports
  156. 156. Learn more... • Book: Interviewing Users by Steve Portigal • Book: Moderating Usability Tests by Joe Dumas & Beth Loring • Book: Paper Prototyping (chapter on task design) by Caroline Snyder • Book: Rocket Surgery Made Easy by Steve Krug • Book:The Moderator’s Survival Guide by Donna Tedesco & Fiona Tranquada
  157. 157. • Article: Seven Common Usability Testing Mistakes by Jared Spool http://www.uie.com/articles/usability_testing_mistakes/ • Article: Interview-Based Tasks: Learning from Leonardo DiCaprio by Jared Spool http://www.uie.com/articles/interview_based_tasks/ • Article: Mobile Testing Toolbox: Part 1&2 by Amber Derosa http://uxmag.com/articles/mobile-testing-toolbox-part-1 • Article: How To Conduct A Usability Test On A Mobile Device by Jeff Sauro http://www.measuringusability.com/blog/mobile-usability-test.php • Article:Why And How To SegmentYour Customers by Jeff Sauro http://www.measuringu.com/blog/segment-customers.php
  158. 158. www.usableinterface.com @kylesoucy Thank you. /usableinterface

×