Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Unmoderated Remote Usability Testing: Good or Evil?


Published on

Recently, there has been a surge in the number of tools that are available to conduct unmoderated (“automated” or “asynchronous”) remote usability testing. This surge is changing the user experience industry and it forces us, whether we want to or not, to take a closer look at what the benefits and drawbacks are of unmoderated testing and whether or not we should incorporate it into our usability toolbox.

In this session we will cover: what you can learn from unmoderated testing, how actionable the data is, how it’s conducted, when it should be conducted, benefits and drawbacks, and an overview of some unmoderated testing tools that are currently available. If there is time, a discussion with attendees about their experience with unmoderated testing will also take place. The content of this session is aimed at anyone who conducts usability testing or is interested in it. The material is good for both the novice and advanced UX practitioner.

The content in this session is aimed at being 100% practical and applicable. Attendees should be able to immediately apply what they learn in this session to evaluate whether or not unmoderated usability testing is a good fit for them. This session will also give attendees the information they need to start conducting their own unmoderated tests. The material that will be covered in this session is an extension of an article that was written by the presenter for

Published in: Design, Technology

Unmoderated Remote Usability Testing: Good or Evil?

  1. 1. Unmoderated Remote Usability Testing: ? Kyle Soucy @usableinterface Boston UPA Mini Conference 2010
  2. 2. What is Unmoderated Remote Testing?
  3. 3. Unmoderated Testing Tools (There are a lot!)
  4. 4. We Won’t Talk About...
  5. 5. We Will Talk About...
  6. 6. We Will Talk About...
  7. 7. Has Anyone Here Used These Tools Before?
  8. 8. What You Can Learn (Depends on the tool and how you use it!)
  9. 9. Web Analytics Image source:
  10. 10. Task Completion Rates Image source: Loop11
  11. 11. Number of Clicks/ Clickstream Image source: Userlytics
  12. 12. Time on Task/Page Image source:
  13. 13. Satisfaction Ratings/ Opinion Rankings It is simple to use: Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree Image source:
  14. 14. True Intent Image source:
  15. 15. In-context Feedback Image source: Loop11
  16. 16. In-context Feedback “...I’m only offered a search by model number. How ridiculous.” Image source: Loop11
  17. 17. In-context Feedback Image source: Loop11
  18. 18. In-context Feedback “...I would like to see the products availability and delivery/ shipping information listed.” Image source: Loop11
  19. 19. In-context Feedback Image source: Loop11
  20. 20. Conducting Unmoderated Remote Usability Tests
  21. 21. Process 1. Define the study 2. Recruit participants 3. Launch test and send email invites 4. Analyze results
  22. 22. Process 3. Ru na 1. Define the study Pilo 2. Recruit participants t tes > t! 4 / Launch test and send email invites 3. 5 / Analyze results 4.
  23. 23. Benefits
  24. 24. Test hundreds of people simultaneously while keeping them in their own natural environment.
  25. 25. Test multiple websites simultaneously.
  26. 26. Reduced costs... depending on the tool you use.
  27. 27. Great way to plant the seed of UCD methodologies and introduce usability testing into a company. UX
  28. 28. Fewer logistics to manage and no need to worry about no-shows and getting last-minute replacements.
  29. 29. Drawbacks
  30. 30. Nothing beats watching participants in real time and being able to ask probing questions about what they are doing as it’s happening. Image source: facit digital
  31. 31. Some participants may only be interested in earning the honorarium you’ve provided as an incentive. $$$
  32. 32. You cannot conduct interview-based tasks.
  33. 33. What participants report on surveys can be very different in comparison to what they actually do. It is simple to use: Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree Image source:
  34. 34. It’s possible for participants to think they’ve successfully completed a task when they haven’t. Image source:
  35. 35. Does it matter if it took someone longer to complete a task? Image source:
  36. 36. When to Conduct Automated Testing?
  37. 37. Not a replacement for moderated testing or research. Best used to compliment your qualitative research.
  38. 38. Unmoderated Testing Tools (A closer look...)
  39. 39. Unmoderated Testing Tools (A closer look...)
  40. 40. Automated Testing Tools with Recruiting Panels
  41. 41. The danger... I wonder how a real customer would actually use this website? Panelist
  42. 42. Video
  43. 43. Video
  44. 44. The danger... BLOCKED BLOCKED
  45. 45. There are no shortcuts to conducting good research.
  46. 46. Automated Testing Tools with Other Recruiting Options
  47. 47. Loop11
  48. 48. Treejack
  49. 49. Chalkmark
  50. 50. Chalkmark Results
  51. 51. Automated Tools with Audio and Video
  52. 52. Handout: Ov erview of Unmo derated Usability Testing Tools
  53. 53. Resources • • Book: Remote Research by Nate Bolt & Tony Tulathimutte • Article: Unmoderated, Remote Usability Testing: Good or Evil? by Kyle Soucy testing-good-or-evil.php
  54. 54. @usableinterface