Unmoderated Remote Usability Testing: Good or Evil?


Published on

Recently, there has been a surge in the number of tools that are available to conduct unmoderated (“automated” or “asynchronous”) remote usability testing. This surge is changing the user experience industry and it forces us, whether we want to or not, to take a closer look at what the benefits and drawbacks are of unmoderated testing and whether or not we should incorporate it into our usability toolbox.

In this session we will cover: what you can learn from unmoderated testing, how actionable the data is, how it’s conducted, when it should be conducted, benefits and drawbacks, and an overview of some unmoderated testing tools that are currently available. If there is time, a discussion with attendees about their experience with unmoderated testing will also take place. The content of this session is aimed at anyone who conducts usability testing or is interested in it. The material is good for both the novice and advanced UX practitioner.

The content in this session is aimed at being 100% practical and applicable. Attendees should be able to immediately apply what they learn in this session to evaluate whether or not unmoderated usability testing is a good fit for them. This session will also give attendees the information they need to start conducting their own unmoderated tests. The material that will be covered in this session is an extension of an article that was written by the presenter for UXmatters.com: http://www.uxmatters.com/mt/archives/2010/01/unmoderated-remote-usability-testing-good-or-evil.php

Published in: Design, Technology
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

  • Sometimes called automated testing.

    Not in a lab; remote.

    Automated system gives the tasks and records data.

  • Unmoderated Remote Usability Testing: Good or Evil?

    1. 1. Unmoderated Remote Usability Testing: ? Kyle Soucy @usableinterface Boston UPA Mini Conference 2010
    2. 2. What is Unmoderated Remote Testing?
    3. 3. Unmoderated Testing Tools (There are a lot!)
    4. 4. We Won’t Talk About...
    5. 5. We Will Talk About...
    6. 6. We Will Talk About...
    7. 7. Has Anyone Here Used These Tools Before?
    8. 8. What You Can Learn (Depends on the tool and how you use it!)
    9. 9. Web Analytics Image source: http://www.hiero.com/web-analytics.html
    10. 10. Task Completion Rates Image source: Loop11
    11. 11. Number of Clicks/ Clickstream Image source: Userlytics
    12. 12. Time on Task/Page Image source: http://blog.usabilla.com/underdogs-beat-expedia-in-usability-test/
    13. 13. Satisfaction Ratings/ Opinion Rankings It is simple to use: Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree Image source: http://wikidoc.org/index.php/Pain
    14. 14. True Intent Image source: http://ethnio.com/images/live_screener_example_page1.jpg
    15. 15. In-context Feedback Image source: Loop11
    16. 16. In-context Feedback “...I’m only offered a search by model number. How ridiculous.” Image source: Loop11
    17. 17. In-context Feedback Image source: Loop11
    18. 18. In-context Feedback “...I would like to see the products availability and delivery/ shipping information listed.” Image source: Loop11
    19. 19. In-context Feedback Image source: Loop11
    20. 20. Conducting Unmoderated Remote Usability Tests
    21. 21. Process 1. Define the study 2. Recruit participants 3. Launch test and send email invites 4. Analyze results
    22. 22. Process 3. Ru na 1. Define the study Pilo 2. Recruit participants t tes > t! 4 / Launch test and send email invites 3. 5 / Analyze results 4.
    23. 23. Benefits
    24. 24. Test hundreds of people simultaneously while keeping them in their own natural environment.
    25. 25. Test multiple websites simultaneously.
    26. 26. Reduced costs... depending on the tool you use.
    27. 27. Great way to plant the seed of UCD methodologies and introduce usability testing into a company. UX
    28. 28. Fewer logistics to manage and no need to worry about no-shows and getting last-minute replacements.
    29. 29. Drawbacks
    30. 30. Nothing beats watching participants in real time and being able to ask probing questions about what they are doing as it’s happening. Image source: facit digital
    31. 31. Some participants may only be interested in earning the honorarium you’ve provided as an incentive. $$$
    32. 32. You cannot conduct interview-based tasks.
    33. 33. What participants report on surveys can be very different in comparison to what they actually do. It is simple to use: Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree Image source: http://www.hiero.com/web-analytics.html
    34. 34. It’s possible for participants to think they’ve successfully completed a task when they haven’t. Image source: http://www.hiero.com/web-analytics.html
    35. 35. Does it matter if it took someone longer to complete a task? Image source: http://www.hiero.com/web-analytics.html
    36. 36. When to Conduct Automated Testing?
    37. 37. Not a replacement for moderated testing or research. Best used to compliment your qualitative research.
    38. 38. Unmoderated Testing Tools (A closer look...)
    39. 39. Unmoderated Testing Tools (A closer look...)
    40. 40. Automated Testing Tools with Recruiting Panels
    41. 41. The danger... I wonder how a real customer would actually use this website? Panelist
    42. 42. UserTesting.com Video
    43. 43. UserTesting.com Video
    44. 44. The danger... BLOCKED BLOCKED
    45. 45. There are no shortcuts to conducting good research.
    46. 46. Automated Testing Tools with Other Recruiting Options
    47. 47. Loop11
    48. 48. Treejack
    49. 49. Chalkmark
    50. 50. Chalkmark Results
    51. 51. Automated Tools with Audio and Video
    52. 52. Handout: Ov erview of Unmo derated Usability Testing Tools
    53. 53. Resources • RemoteUsability.com • Book: Remote Research by Nate Bolt & Tony Tulathimutte • Article: Unmoderated, Remote Usability Testing: Good or Evil? by Kyle Soucy http://www.uxmatters.com/mt/archives/2010/01/unmoderated-remote-usability- testing-good-or-evil.php
    54. 54. www.usableinterface.com @usableinterface
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.