Your SlideShare is downloading. ×
0
Webinar:
How to Conduct Unmoderated Remote
Usability Testing
Intros
Alfonso de la Nuez
Co-Founder & Co-CEO
UserZoom
Nate Colker
UX Researcher
UserZoom
John Romadka
UX Researcher
Optum...
Overview
Part 1 (20 min)
 Introduction to unmoderated remote usability testing
 A real life case study featuring a compa...
User testing, feedback & analytics tools mania!
Let’s try to make some sense out of this…
Many ways of organizing remote testing tools:
 Moderated vs. Unmoderated
 Qual...
Remote, unmoderated - A growing research method
Why this testing tool fever?
 Consumers more empowered than ever, thx in part to Social Media
 Ecommerce maturing, more ...
Why Should You Care about Remote Testing?
 Cost-effective, save time and money on research
 You can quantify usability, ...
What can be tested with URUT?
 Static mockups
 Wireframes
 Prototypes
 Multimedia
 Live websites
 Intranets
 …any w...
Pros & Cons
Pros:
 Cost-effectiveness
 Quantify UX
 Geographic representation
 Honest feedback
 Combination of result...
Case Study:
Remote UX Studies - Tools & Methods
John Romadka, UX Researcher
OptumInsight (Division of UnitedHealth Group)
Challenges
New project
We’d depleted our pool of internal (agent) participants
locally
Needed to get a broader perspective...
Challenges
Couldn’t convince business partners to test iteratively
Lab studies were costly
and often resulted in
large mar...
Making the Case
Made the unmoderated testing business case
More cost efficient
• Unmoderated testing can cost 50% less tha...
Making the Case
Validation Testing
Retest within days, instead of week
Moderated test: 180 hours vs. Unmoderated test: 40 ...
Results
Successful company-wide adoption of UserZoom
Obtained a pilot group of “Field” participants
Broad range of demogra...
Results
UserZoom allowed us to quickly setup/run/analyze
Online Surveys
Task-based studies
Card Sort exercises
Tree Testin...
Results
Task Based Studies -
Remote Moderated: combine 3 tools
UserZoom (metrics)
Lead users through study
Central point f...
Results
Task Based Studies -
Combined Unmoderated/Moderated
Users were onsite occasionally
Validate our unmoderated result...
Thank You!
@jaromadGplus.to/jromadka
How to Conduct Unmoderated Remote Usability Testing
• How to plan for an unmoderated remote usability study
• What metrics...
What are the goals of the study?
A Few Examples….
Evaluate and Quantify the User Experience
Comparing Multiple Designs (A/...
What are the goals of the study?
A Few Examples….
Remote Task-Based
True Intent
Card Sort / Tree Test
Evaluate and Quantif...
A Few Examples….
Remote Task-Based
True Intent
Card Sort / Tree Test
Evaluate and Quantify the User Experience
Comparing M...
A Hulu Demo Study
Planning Design Recruitment Analysis
What Metrics do you Need to Collect?
 Jeff Sauro,
Measuringusability.com
Planning Design Recruitment Analysis
Create A Study Script
Recruitment AnalysisPlanning Design
• Study Goals & Participant Profile
• Welcome Page
• Initial Que...
Welcome Page Design
Recruitment AnalysisPlanning Design
Set Expectations for the participants
• Introduce study
• Time est...
Example Welcome Page
Recruitment AnalysisPlanning Design
Initial Questions - Tips
Recruitment AnalysisPlanning Design
You may want to
collect what type of
experience users
have wi...
Initial Question Examples
Recruitment AnalysisPlanning Design
Initial Question – Logic & Conditions
Recruitment AnalysisPlanning Design
• Skip participants to a specific task or questi...
Selecting the Right Tasks
Recruitment AnalysisPlanning Design
Balance of User Data and Business Goals
• What are users com...
Selecting the Right Tasks
Recruitment AnalysisPlanning Design
Balance of User Data and Business Goals
• What are users com...
Task Construction - Tips
Recruitment AnalysisPlanning Design
Tasks should be
natural and easy to
understand, but not
easy ...
Building Multiple Tasks - Randomizing Order
Recruitment AnalysisPlanning Design
Screener Pre-task questions
Hulu Task 1
Hu...
Task Validation (3 options)
Recruitment AnalysisPlanning Design
• Validate by question
• “Quiz question” used to verify ta...
Task Description Example
Recruitment AnalysisPlanning Design
Post Task Questions
Recruitment AnalysisPlanning Design
Self-Reported Data
• Are user satisfied with the experience?
• Do ...
Post Task Question Examples
Recruitment AnalysisPlanning Design
Validation
Question
Error
Question
Success
Final Questionnaire - Tips
Recruitment AnalysisPlanning Design
Overall, how easy or
difficult was it to use
this website?
...
Final Questionnaire Example Questions
Recruitment AnalysisPlanning Design
What about recruiting participants?
AnalysisPlanning Design Recruitment
Define a screener
Hulu Demo Study
• Representative...
What about recruiting participants?
AnalysisPlanning Design Recruitment
Quality Control Settings
AnalysisPlanning Design Recruitment
Quality Controls
• Be careful not to exclude fast users
• Che...
What about recruiting participants?
 3 choices to recruit participants:
1. Panel vendor (SSI, Research Now, etc)
2. Priva...
Hulu Demo Walkthrough
Planning Design Recruitment Analysis
Analyzing the Results
Planning Design Recruitment Analysis
1. Task Success
2. Efficiency (time, clicks)
3. Self-Reported S...
Analyzing the Results – Clean Data
Planning Design Recruitment Analysis
Review your data for outliers / “bad data”
• Very ...
Make assumptions about user population (Confidence Intervals)
Hulu baselines can be compared with data from future study (...
Analyzing Metrics – Open Ended
Planning Design Recruitment Analysis
Use Text Clouds & Search for Patterns
Use Comment Codi...
Analyzing Metrics – Behavioral
Planning Design Recruitment Analysis
What strategies are being used (how successful are the...
Q&A
Thanks so much for your time!
Check out our upcoming news and events at www.userzoom.com
Sunnyvale (USA)
440 N. Wolfe Rd. ...
Upcoming SlideShare
Loading in...5
×

Webinar: How to Conduct Unmoderated Remote Usability Testing

1,619

Published on

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,619
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
56
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Saves time – Very fast, thousands on panels, Money – essence of quick and dirty. Techniques for dealing with noise, unrealistic to be in the lab that long. Combines both qual/quant and attitudes and behavior.
  • Thanks going to cover the process of creating a study and the steps involvedBraking down in 4 topic areas
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesIllustrate again very large space. We’re just going to talk about this
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesIllustrate again very large space. We’re just going to talk about this
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesEvaluate and Quantify UX  Baseline study we can compare results as we redesign or over timeTask-Based users going to be navigation on websiteIllustrate again very large space. We’re just going to talk about this
  • Disclaimer : not a customer just a demo example
  • Our Goal  Evaluation and Quantifying the UX Baseline Test we can compare our results laterPartial list from measuring usability (there are several of these)Completion Rate  providing tasks to users, if they can’t complete the tasks not much else matters, going to be a primary focus.We also have a high sample size. Important for analysis. we’re going to be able to make better predictions than what you would get in a lab study with 3-5 users.We’re also going to be looking at efficiency(task time), satisfaction. Triangulate data. Tell a more powerful storyBehavioral data too. Where did people go and what did they click Plan on what metrics you want and know how you can use them.
  • Lets get started We need to have a plan  here is a simple outlineWe have a welcome pageInitial questionsTasks where users are going to be sent to the Hulu websiteQuestions specific to those tasksFinal questionsGonna walkthrough each of these areas
  • Set Expectations
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Gather feedback on the holistic experiencePre-post metrics
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • Disclaimer : not a customer just a demo example
  • Success are people able to completeEfficiency not as crucial for Hulu depends on your domain. We can identify problem areas
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • Transcript of "Webinar: How to Conduct Unmoderated Remote Usability Testing"

    1. 1. Webinar: How to Conduct Unmoderated Remote Usability Testing
    2. 2. Intros Alfonso de la Nuez Co-Founder & Co-CEO UserZoom Nate Colker UX Researcher UserZoom John Romadka UX Researcher Optum Insight (Division of UnitedHealth Group)
    3. 3. Overview Part 1 (20 min)  Introduction to unmoderated remote usability testing  A real life case study featuring a company in the insurance industry Part 2 (30 min)  How to plan for an unmoderated remote usability study  How to design a study  What metrics you need to collect  How to recruit participants  How to analyze the data  Q&A
    4. 4. User testing, feedback & analytics tools mania!
    5. 5. Let’s try to make some sense out of this… Many ways of organizing remote testing tools:  Moderated vs. Unmoderated  Qualitative vs. Quantitative-based  User engaging vs. Non-user engaging (analytics)  Usability focused vs. ‘quick user feedback’ (VOC)  Video vs. Stats/metrics  Etc.
    6. 6. Remote, unmoderated - A growing research method
    7. 7. Why this testing tool fever?  Consumers more empowered than ever, thx in part to Social Media  Ecommerce maturing, more competition  Web analytics, usability test in the lab, mkt research… no longer enough or are very costly  Agile dev is all about speed, iteration 1. Designing & Testing UX quickly & cost-effectively is KEY 2. People want better!
    8. 8. Why Should You Care about Remote Testing?  Cost-effective, save time and money on research  You can quantify usability, obtain statistically significant UX data  Run competitive UX Benchmarking  Iterative testing, validate design/prototypes  Test with users from different geographic locations  Objectivity + accountability  Improves communication  Mitigate risks
    9. 9. What can be tested with URUT?  Static mockups  Wireframes  Prototypes  Multimedia  Live websites  Intranets  …any web-based UI  …anything you can host on a server and access through a web browser
    10. 10. Pros & Cons Pros:  Cost-effectiveness  Quantify UX  Geographic representation  Honest feedback  Combination of results and metrics Cons:  Obviously… no face to face
    11. 11. Case Study: Remote UX Studies - Tools & Methods John Romadka, UX Researcher OptumInsight (Division of UnitedHealth Group)
    12. 12. Challenges New project We’d depleted our pool of internal (agent) participants locally Needed to get a broader perspective from agents across country Network security issues Limited number of dedicated UX Researchers
    13. 13. Challenges Couldn’t convince business partners to test iteratively Lab studies were costly and often resulted in large margin of error 90% confidence interval
    14. 14. Making the Case Made the unmoderated testing business case More cost efficient • Unmoderated testing can cost 50% less than traditional moderated testing • Primarily due to reduction in effort hours spent in lab Timeline • Reduced the 3 week lead time for recruiting to less than 1 week.
    15. 15. Making the Case Validation Testing Retest within days, instead of week Moderated test: 180 hours vs. Unmoderated test: 40 hours Remote Unmoderated Testing performed for ~20% the cost of benchmark Moderated test Test 1- (Moderated) = $10,000 Test 2- (Unmoderated) = $2000) Improved Confidence Intervals
    16. 16. Results Successful company-wide adoption of UserZoom Obtained a pilot group of “Field” participants Broad range of demographics, locations Bi-weekly expectation of participating in whatever UX research was available and ready to study • Required less context, setup
    17. 17. Results UserZoom allowed us to quickly setup/run/analyze Online Surveys Task-based studies Card Sort exercises Tree Testing studies Task-based studies: Moderated Remote Unmoderated Remote Moderated Lab
    18. 18. Results Task Based Studies - Remote Moderated: combine 3 tools UserZoom (metrics) Lead users through study Central point for metrics Quick and easy to re-test Webex (screen sharing, audio) Deployed company-wide Giving participant control of web app on dev Morae (note taking, recording)
    19. 19. Results Task Based Studies - Combined Unmoderated/Moderated Users were onsite occasionally Validate our unmoderated results UserZoom: Used same setup for both Consistent test method and process Freed up the Facilitator to observe, take notes. Knowing UserZoom was gathering metrics
    20. 20. Thank You! @jaromadGplus.to/jromadka
    21. 21. How to Conduct Unmoderated Remote Usability Testing • How to plan for an unmoderated remote usability study • What metrics you need to collect • How to design a study • How to recruit participants • How to analyze the data Planning Design Recruitment Analysis
    22. 22. What are the goals of the study? A Few Examples…. Evaluate and Quantify the User Experience Comparing Multiple Designs (A/B) Competitive Benchmark Identify Visitors’ Core Tasks Redesign Navigation Structure Improve the Navigation Experience Planning Design Recruitment Analysis
    23. 23. What are the goals of the study? A Few Examples…. Remote Task-Based True Intent Card Sort / Tree Test Evaluate and Quantify the User Experience Comparing Multiple Designs (A/B) Competitive Benchmark Identify Visitors’ Core Tasks Redesign Navigation Structure Improve the Navigation Experience Planning Design Recruitment Analysis Method
    24. 24. A Few Examples…. Remote Task-Based True Intent Card Sort / Tree Test Evaluate and Quantify the User Experience Comparing Multiple Designs (A/B) Competitive Benchmark Identify Visitors’ Core Tasks Redesign Navigation Structure Improve the Navigation Experience Planning Design Recruitment Analysis Method What are the goals of the study?
    25. 25. A Hulu Demo Study Planning Design Recruitment Analysis
    26. 26. What Metrics do you Need to Collect?  Jeff Sauro, Measuringusability.com Planning Design Recruitment Analysis
    27. 27. Create A Study Script Recruitment AnalysisPlanning Design • Study Goals & Participant Profile • Welcome Page • Initial Questions • Task 1 • Post-Task Questions • Task 2 • Post-Task Questions • Final Questionnaire
    28. 28. Welcome Page Design Recruitment AnalysisPlanning Design Set Expectations for the participants • Introduce study • Time estimate and incentive • Keep short and to the point Purpose of the study Time estimate Incentive Technical requirements Instructions
    29. 29. Example Welcome Page Recruitment AnalysisPlanning Design
    30. 30. Initial Questions - Tips Recruitment AnalysisPlanning Design You may want to collect what type of experience users have with the product. You can use branching logic to give certain tasks to specific participants based on their expertise Avoid asking hypothetical questions. Participants are not good at knowing how they’ll behave in a certain situation. “Would you book flights on this website if we added this option?” is a bad example Be specific. “Daily” and “Weekly” is more specific than “Always” and “Sometimes”. Avoid “double questions” that have two questions in one. “Please indicate how easy and fast it was for you to complete the task?” Avoid asking yes/no questions. It can lead participants into an extreme statement. “Do you use this website all the time?” is not a good example. Don’t reveal too much information to the participant upfront Brand recognition can have a large impact on people’s reactions to products.
    31. 31. Initial Question Examples Recruitment AnalysisPlanning Design
    32. 32. Initial Question – Logic & Conditions Recruitment AnalysisPlanning Design • Skip participants to a specific task or question • Hide or show questions based on answers (example above) • Disqualify participants
    33. 33. Selecting the Right Tasks Recruitment AnalysisPlanning Design Balance of User Data and Business Goals • What are users coming to your site to do (core tasks) ? • Are there any tasks that are required to use your site? • What drives revenue on your site?
    34. 34. Selecting the Right Tasks Recruitment AnalysisPlanning Design Balance of User Data and Business Goals • What are users coming to your site to do (core tasks) ? - Find and watch a TV show • Are there any tasks that are required to use your site? - Create an account? • What drives revenue on your site? - Sign up for a premium account
    35. 35. Task Construction - Tips Recruitment AnalysisPlanning Design Tasks should be natural and easy to understand, but not easy to guess Randomize task order to minimize the learning effect, if tasks are not dependent on each other. Create tasks that are easy to validate (you need to be able to confirm that the participant actually found an answer). Tasks should be worded in a way that doesn’t lead the participant When using Validation word tasks so that there can be a discrete answer (price, date etc.) Include “none of the above” option (it will discourage participants from using deduction to figure out the answer).
    36. 36. Building Multiple Tasks - Randomizing Order Recruitment AnalysisPlanning Design Screener Pre-task questions Hulu Task 1 Hulu Task 2 Control for learning effects • Participants become more proficient as they use the site There are exceptions, sometimes tasks must be completed in order Final QuestionsRandomized
    37. 37. Task Validation (3 options) Recruitment AnalysisPlanning Design • Validate by question • “Quiz question” used to verify task completion • Validate by URL • Successful if participant reach a specific URL • Think carefully about task wording and defining success • “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search results page vs. the specific episode page) • Self-reported
    38. 38. Task Description Example Recruitment AnalysisPlanning Design
    39. 39. Post Task Questions Recruitment AnalysisPlanning Design Self-Reported Data • Are user satisfied with the experience? • Do users believe they completed the task successfully? • Why wasn’t a participant successful ? Task Success Button Validation Abandon Button Abandon Questions Error Questions Success Questions
    40. 40. Post Task Question Examples Recruitment AnalysisPlanning Design Validation Question Error Question Success
    41. 41. Final Questionnaire - Tips Recruitment AnalysisPlanning Design Overall, how easy or difficult was it to use this website? How likely is it that you would recommend this website to a friend or colleague? After your experience with the website today, which of the following words do you associate with the brand? When using Validation word tasks so that there can be a discrete answer (price, date etc.) OVERALL RATING SCALES NET PROMOTER SCORE CHANGE IN BRAND PERCEPTION OPEN-ENDED QUESTIONS
    42. 42. Final Questionnaire Example Questions Recruitment AnalysisPlanning Design
    43. 43. What about recruiting participants? AnalysisPlanning Design Recruitment Define a screener Hulu Demo Study • Representative users • Demographics (gender, age etc.) • Control for experience • Larger sample size -> more precise predictions • Set profiles and define quota limits • Set quality controls
    44. 44. What about recruiting participants? AnalysisPlanning Design Recruitment
    45. 45. Quality Control Settings AnalysisPlanning Design Recruitment Quality Controls • Be careful not to exclude fast users • Cheaters often take the most efficient path (look out for “0 clicks”)
    46. 46. What about recruiting participants?  3 choices to recruit participants: 1. Panel vendor (SSI, Research Now, etc) 2. Private Mailing List 3. Intercept visitors AnalysisPlanning Design Recruitment
    47. 47. Hulu Demo Walkthrough Planning Design Recruitment Analysis
    48. 48. Analyzing the Results Planning Design Recruitment Analysis 1. Task Success 2. Efficiency (time, clicks) 3. Self-Reported Satisfaction 4. Open prompts 5. Behavioral Data
    49. 49. Analyzing the Results – Clean Data Planning Design Recruitment Analysis Review your data for outliers / “bad data” • Very High / Very Low task times • 0 Clicks • Nonsense Answers
    50. 50. Make assumptions about user population (Confidence Intervals) Hulu baselines can be compared with data from future study (Significance Testing) Analyzing Metrics – Beyond the Tool Planning Design Recruitment Analysis Jeff Sauro, StatUsabilityPak
    51. 51. Analyzing Metrics – Open Ended Planning Design Recruitment Analysis Use Text Clouds & Search for Patterns Use Comment Coding for more Detailed Analysis
    52. 52. Analyzing Metrics – Behavioral Planning Design Recruitment Analysis What strategies are being used (how successful are they?) What are “Non-Success” participants doing? (filter data)
    53. 53. Q&A
    54. 54. Thanks so much for your time! Check out our upcoming news and events at www.userzoom.com Sunnyvale (USA) 440 N. Wolfe Rd. Sunnyvale, CA 94085 Phone: +1 (408) 524 7445 Contact: Alfonso de la Nuez Barcelona (Spain) Av. Diagonal 419 3º 2ª 08008 Barcelona Phone: +34 93 414 7554 Fax: +34 93 209 8380 Contact: Xavier Mestres Cheshire (UK) Booths Park. Chelford Road, Knutsford, Cheshire WA16 8GS Phone: + 44 (0)1565 759890 Cell: + 44 (0)7900 472 920 Contact: Arthur Moan
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×