Remote Testing Methods & Tools Webinar


Published on

On Tuesday, December 13th, Bill Albert, co-author of Beyond the Usability Lab, and one of the most knowledgable experts in Remote Usability Testing, presented an update on Remote User Testing methods and tools to optimize your UX Roadmap. If you were not able to attend the live webinar please feel free to view the slideshare below!

The webinar was co-hosted with 3 special guests:

Dave Garr, Co-Founder,

Andrew Mayfield, CEO, Optimal Workshop

Matt Paulus, Director of US Sales, UserZoom

Published in: Education, Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Introduce myself and DUC Feel free to ask questions Motivation – sort through powerful tools, trying to find the right ones for the right job
  • Where it fits in Highly adaptive based on the needs This gives it POWER – it creates the missing link
  • Saves time – Very fast, thousands on panels, Money – essence of quick and dirty. Techniques for dealing with noise, unrealistic to be in the lab that long. Combines both qual/quant and attitudes and behavior.
  • All the flexibility you need to set up a study and analyzing the dataSignificant support in designing study and analysis. Pricing is all project based – typically very expensive – good choice for a large benchmark study
  • - These tools are self-service, with some amount of supportCan do largely the same as RV and Keynote, and becoming more powerful by the day Nice visualizations now Loop 11 is very basic, and UZ is much more flexible fraction of the cost = several hundred to a few thousand.
  • - Sort into groups
  • You get a video (typically about 15 minutes)This is a test of and her task is to find an ink cartridge for her printerShe doesn’t know what’s going on, and she doesn’t know how to fix it.
  • She spends more than four minutes and can’t figure out which cartridge works with her (very popular) HP deskjet 1000 printer.
  • She spends more than four minutes and can’t figure out which cartridge works with her (very popular) HP deskjet 1000 printer.
  • Download a clip and just drag it into Powerpoiont.
  • Download a clip and just drag it into Powerpoiont.
  • Download a clip and just drag it into Powerpoiont.
  • Download a clip and just drag it into Powerpoint.
  • Remote Testing Methods & Tools Webinar

    1. 1. Bill Albert, Director of Design & Usability Center at Bentley University presents a complimentary webinar on Remote Testing Methods and Tools.With special guests: Webinar: Remote Testing Methods & Tools December 13th, 2011
    2. 2. Webinar Overview1. Brief introductions & goal of the webinar2. Updated overview of competitive landscape, updated figures, etc3. The different methods / categories / tools. When to use what?4. Types of research goals 1. Categories of tools available 2. Overview of some of those tools5. Examples: – – Optimal Workshop – UserZoom6. Takeaways, Q&A 2
    3. 3. Introduction • Speakers today: o Bill Albert, Director of Design & Usability Center, Bentley University o Dave Garr, Co-Founder, o Andrew Mayfield, CEO, Optimal Workshop o Matt Paulus, Director of US Sales, UserZoom • Goals of today’s session 3
    4. 4. Remote Testing Methods and Tools Bill Albert December 13, 2011 781.891.2500 |
    5. 5. Agenda• Introduction to remote usability testing• Quantitative-based tools• Qualitative-based tools 5
    6. 6. About me• Director, Design and Usability Center, Bentley University• Experience with remote testing tools• I am data driven! 6
    7. 7. I am a big fan of usability testing in the lab! 7
    8. 8. Introduction to remote usability testing
    9. 9. Defining Remote Testing ToolsWhat do we mean by “remote testing tools”?• A remote testing tool is any technology that allows a researcher to collect data from users about their experience in using an interface, without direct, face-to-face contact.There are two ways of organizing remote testing tools• Moderated vs. Unmoderated• Qualitative vs. Quantitative-based Moderated Unmoderated Qualitative-based Qualitative-based Quantitative-based Quantitative-based 9
    10. 10. Unmoderated Testing Tools Full-service Self-service Card Quantitative- Sorting/IA based Unmoderated Surveys tools Click/Mouse Video tools Qualitative- based Reporting Expert Reviews 10
    11. 11. Our UX Toolkit 11
    12. 12. Why Should You Care?• Saves time o Lab study takes 2-4 weeks from start to finish, unmoderated typically takes hours to a few days*• Saves money o Participants compensation typically a lot less ($10 vs. $100) o Tools are becoming very inexpensive• Reliable metrics o Only (reasonable) way to collect UX data from large sample sizes• Geography is not a limitation o Collect feedback from customers all over the world• Greater Customer insight o Richest dataset about the customer experience 12
    13. 13. Why is this important NOW?• Questions from senior management are becoming more complex and time sensitive• Traditional usability is no longer enough• Push to measure the UX• Convergence with market research and web analytics to paint a more complete picture of the UX• Budgets/timeline constraints 13
    14. 14. Which one goes first? Lab first, then Unmoderated Unmoderated first, then Lab Identify/fix “low hanging fruit”, then Identify the most significant issues focus on remaining tasks with large online through metrics, then use lab sample size study to gather deeper qualitative understanding of those issues Generate new concepts, ideas, Collect video clips or more quotes of questions through lab testing, then users to help bring metrics to life test/validate online Validate attitudes/preferences Gather all the metrics to validate observed in lab testing design – if it tests well, then no need to bring users into the lab 14
    15. 15. Quantitative-based tools
    16. 16. Unmoderated Testing Tools - Quantitative Full-service Self-service Card Quantitative- Sorting/IA based Unmoderated Surveys tools Click/Mouse Video tools Qualitative- based Reporting Expert Reviews 16
    17. 17. OverviewCommon Research Questions:• What are the usability issues, and how big?• Which design is better, and by how much?• How do customer segments differ?• What are user design preferences?• Is the new design better than the old design?• Where are users most likely to abandon a transaction?Types of Studies: Typical Metrics:• Comprehensive evaluation • Task success• UX benchmark • Task time• Competitive evaluation • Self-report ratings such as ease of• Live site vs. prototype comparison use, confidence, satisfaction• Feature/function test • SUS• Discovery • Click paths • Abandonment 17
    18. 18. Strengths / Limitations / MythsStrengths:• Comparing products• Measuring user experience• Finding the right participants• Focusing on design improvements• Insight into the user’s real experienceLimitations: Myths:• Not well suited to rapid, iterative • Only test with websites design • Very expensive• Need a deep understanding of • Only gather quantitative data issues • A lot of noise in the data• Studies that require long sessions • Same as any market research study• Lose control over prototypes • Comparing products• Internet access 18
    19. 19. Full Service Tools – Online Usability Testing 19
    20. 20. Self Service Tools – Online Usability Testing 20
    21. 21. Card Sorting / IA Tools 21
    22. 22. Online Surveys 22
    23. 23. Click and Mouse Tools 23
    24. 24. Qualitative-based tools
    25. 25. Unmoderated Testing Tools - Qualitative Full-service Self-service Card Quantitative- Sorting/IA based Unmoderated Surveys tools Click/Mouse Video tools Qualitative- based Reporting Expert Reviews 25
    26. 26. OverviewCommon Research Questions:• What are the big pain usability pain points, and wins?• How do users think, and feel about the design?• Are we on the right (design) track?• What is the overall experience like?• Why might users abandon a transaction?Types of Studies: Common Metrics:• Low fidelity prototype • Frequency of issues• High fidelity prototype • Verbatim comments• Competitive evaluation • Task success (via video)• Comprehensive evaluation • Task time (via video)• Feature/function test• Discovery 26
    27. 27. Video Tools 27
    28. 28. Report Tools 28
    29. 29. Expert Review 29
    30. 30. Additional ResourcesOnline:www.measuringUX.comwww.measuringusability.com me:@UXMetrics@BentleyDUCLinkedIn Group:Bentley Design andUsability Center 30
    31. 31. Dave Garr
    32. 32. What’s your visitor thinking?In Cyberspace, no one can hear your visitor scream.
    33. 33. Solution? 33
    34. 34. How It Works
    35. 35. “I don’t know why it keeps doing this.This is definitely very confusing and frustrating.”
    36. 36. Write an Annotation
    37. 37. “I can’t figure out which cartridge goes with my printer”
    38. 38. How to share clips
    39. 39. View Tester’s Answers
    40. 40. Follow up with users
    41. 41. Export to Excel
    42. 42. Mobile Testing• Test mobile apps• Test mobile-friendly sites 42
    43. 43. “I’d like to look at these three star reviews, … it doesn’t let me do that… I’d have to go through all of the reviews.” What do you want to test? 43
    44. 44. Presenting Your Findings 44
    45. 45. Andrew Mayfield CEOOptimal Workshop
    46. 46. Quantitative analysis to assess the effectiveness of an Information Architecture ishard work.
    47. 47. Peter Morville’sUX Honeycomb
    48. 48. Optimal Workshop
    49. 49. The Suiteby Optimal Workshop
    50. 50. Qualitative vs Quantitative See latest blog post on this
    51. 51. Participant Centric Analysis
    52. 52. Remote studies enable near unlimitediteration and participation BENCHMARK RadicalSafe or optionexistingoptions ?
    53. 53. Matt PaulusDirector of US Sales UserZoom
    54. 54. UserZoom’s Software Solutions • On-demand, web-based SaaS • 100% self-service • All-in-one quantitative unmoderated online research • Enterprise level software • Feature rich & intuitive
    55. 55. Unmoderated Remote Testing Methodology • Quantitative remote testing • Natural context • International/domestic testing • No moderation needed • Behavior (analytics) + Feedback (online Surveys)
    56. 56. UserZoom Products
    57. 57. UserZoom ProductsLive Websites Wireframes & Prototypes Compatible PlatformsMultimedia Mobile Interfacescontent & Apps 60
    58. 58. UserZoom Task-Based Usability Testing
    59. 59. UserZoom Results – Open Ended/Text Clouds
    60. 60. UserZoom Results – Net Promotor Score
    61. 61. UserZoom Results – Filters/Segmentation
    62. 62. UserZoom Results – Effectiveness & Efficiency
    63. 63. UserZoom Results – Clickstreams
    64. 64. UserZoom Results- Click Heatmaps
    65. 65. Key Takeaways• Remote testing tools are becoming more advanced every day, and are a critical part of the UX toolkit• Find the right type of tool based on your research question• Demo a select number of remote testing tools to find the one that will deliver the results you need, within budget and timeframe 68
    66. 66. Quick Poll! 69
    67. 67. Thank You! Bill Albert Dave Garr Andrew Mayfield Matt Paulus Director of Design Co-Founder CEO Director of US & Usability Center Optimal Sales Bentley University Workshop UserZoom Floor open to Q&A