• Like

Webinar: How to Conduct Unmoderated Remote Usability Testing

  • 1,403 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,403
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
55
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Saves time – Very fast, thousands on panels, Money – essence of quick and dirty. Techniques for dealing with noise, unrealistic to be in the lab that long. Combines both qual/quant and attitudes and behavior.
  • Thanks going to cover the process of creating a study and the steps involvedBraking down in 4 topic areas
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesIllustrate again very large space. We’re just going to talk about this
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesIllustrate again very large space. We’re just going to talk about this
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesEvaluate and Quantify UX  Baseline study we can compare results as we redesign or over timeTask-Based users going to be navigation on websiteIllustrate again very large space. We’re just going to talk about this
  • Disclaimer : not a customer just a demo example
  • Our Goal  Evaluation and Quantifying the UX Baseline Test we can compare our results laterPartial list from measuring usability (there are several of these)Completion Rate  providing tasks to users, if they can’t complete the tasks not much else matters, going to be a primary focus.We also have a high sample size. Important for analysis. we’re going to be able to make better predictions than what you would get in a lab study with 3-5 users.We’re also going to be looking at efficiency(task time), satisfaction. Triangulate data. Tell a more powerful storyBehavioral data too. Where did people go and what did they click Plan on what metrics you want and know how you can use them.
  • Lets get started We need to have a plan  here is a simple outlineWe have a welcome pageInitial questionsTasks where users are going to be sent to the Hulu websiteQuestions specific to those tasksFinal questionsGonna walkthrough each of these areas
  • Set Expectations
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Gather feedback on the holistic experiencePre-post metrics
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • Disclaimer : not a customer just a demo example
  • Success are people able to completeEfficiency not as crucial for Hulu depends on your domain. We can identify problem areas
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population

Transcript

  • 1. Webinar: How to Conduct Unmoderated Remote Usability Testing
  • 2. Intros Alfonso de la Nuez Co-Founder & Co-CEO UserZoom Nate Colker UX Researcher UserZoom John Romadka UX Researcher Optum Insight (Division of UnitedHealth Group)
  • 3. Overview Part 1 (20 min)  Introduction to unmoderated remote usability testing  A real life case study featuring a company in the insurance industry Part 2 (30 min)  How to plan for an unmoderated remote usability study  How to design a study  What metrics you need to collect  How to recruit participants  How to analyze the data  Q&A
  • 4. User testing, feedback & analytics tools mania!
  • 5. Let’s try to make some sense out of this… Many ways of organizing remote testing tools:  Moderated vs. Unmoderated  Qualitative vs. Quantitative-based  User engaging vs. Non-user engaging (analytics)  Usability focused vs. ‘quick user feedback’ (VOC)  Video vs. Stats/metrics  Etc.
  • 6. Remote, unmoderated - A growing research method
  • 7. Why this testing tool fever?  Consumers more empowered than ever, thx in part to Social Media  Ecommerce maturing, more competition  Web analytics, usability test in the lab, mkt research… no longer enough or are very costly  Agile dev is all about speed, iteration 1. Designing & Testing UX quickly & cost-effectively is KEY 2. People want better!
  • 8. Why Should You Care about Remote Testing?  Cost-effective, save time and money on research  You can quantify usability, obtain statistically significant UX data  Run competitive UX Benchmarking  Iterative testing, validate design/prototypes  Test with users from different geographic locations  Objectivity + accountability  Improves communication  Mitigate risks
  • 9. What can be tested with URUT?  Static mockups  Wireframes  Prototypes  Multimedia  Live websites  Intranets  …any web-based UI  …anything you can host on a server and access through a web browser
  • 10. Pros & Cons Pros:  Cost-effectiveness  Quantify UX  Geographic representation  Honest feedback  Combination of results and metrics Cons:  Obviously… no face to face
  • 11. Case Study: Remote UX Studies - Tools & Methods John Romadka, UX Researcher OptumInsight (Division of UnitedHealth Group)
  • 12. Challenges New project We’d depleted our pool of internal (agent) participants locally Needed to get a broader perspective from agents across country Network security issues Limited number of dedicated UX Researchers
  • 13. Challenges Couldn’t convince business partners to test iteratively Lab studies were costly and often resulted in large margin of error 90% confidence interval
  • 14. Making the Case Made the unmoderated testing business case More cost efficient • Unmoderated testing can cost 50% less than traditional moderated testing • Primarily due to reduction in effort hours spent in lab Timeline • Reduced the 3 week lead time for recruiting to less than 1 week.
  • 15. Making the Case Validation Testing Retest within days, instead of week Moderated test: 180 hours vs. Unmoderated test: 40 hours Remote Unmoderated Testing performed for ~20% the cost of benchmark Moderated test Test 1- (Moderated) = $10,000 Test 2- (Unmoderated) = $2000) Improved Confidence Intervals
  • 16. Results Successful company-wide adoption of UserZoom Obtained a pilot group of “Field” participants Broad range of demographics, locations Bi-weekly expectation of participating in whatever UX research was available and ready to study • Required less context, setup
  • 17. Results UserZoom allowed us to quickly setup/run/analyze Online Surveys Task-based studies Card Sort exercises Tree Testing studies Task-based studies: Moderated Remote Unmoderated Remote Moderated Lab
  • 18. Results Task Based Studies - Remote Moderated: combine 3 tools UserZoom (metrics) Lead users through study Central point for metrics Quick and easy to re-test Webex (screen sharing, audio) Deployed company-wide Giving participant control of web app on dev Morae (note taking, recording)
  • 19. Results Task Based Studies - Combined Unmoderated/Moderated Users were onsite occasionally Validate our unmoderated results UserZoom: Used same setup for both Consistent test method and process Freed up the Facilitator to observe, take notes. Knowing UserZoom was gathering metrics
  • 20. Thank You! @jaromadGplus.to/jromadka
  • 21. How to Conduct Unmoderated Remote Usability Testing • How to plan for an unmoderated remote usability study • What metrics you need to collect • How to design a study • How to recruit participants • How to analyze the data Planning Design Recruitment Analysis
  • 22. What are the goals of the study? A Few Examples…. Evaluate and Quantify the User Experience Comparing Multiple Designs (A/B) Competitive Benchmark Identify Visitors’ Core Tasks Redesign Navigation Structure Improve the Navigation Experience Planning Design Recruitment Analysis
  • 23. What are the goals of the study? A Few Examples…. Remote Task-Based True Intent Card Sort / Tree Test Evaluate and Quantify the User Experience Comparing Multiple Designs (A/B) Competitive Benchmark Identify Visitors’ Core Tasks Redesign Navigation Structure Improve the Navigation Experience Planning Design Recruitment Analysis Method
  • 24. A Few Examples…. Remote Task-Based True Intent Card Sort / Tree Test Evaluate and Quantify the User Experience Comparing Multiple Designs (A/B) Competitive Benchmark Identify Visitors’ Core Tasks Redesign Navigation Structure Improve the Navigation Experience Planning Design Recruitment Analysis Method What are the goals of the study?
  • 25. A Hulu Demo Study Planning Design Recruitment Analysis
  • 26. What Metrics do you Need to Collect?  Jeff Sauro, Measuringusability.com Planning Design Recruitment Analysis
  • 27. Create A Study Script Recruitment AnalysisPlanning Design • Study Goals & Participant Profile • Welcome Page • Initial Questions • Task 1 • Post-Task Questions • Task 2 • Post-Task Questions • Final Questionnaire
  • 28. Welcome Page Design Recruitment AnalysisPlanning Design Set Expectations for the participants • Introduce study • Time estimate and incentive • Keep short and to the point Purpose of the study Time estimate Incentive Technical requirements Instructions
  • 29. Example Welcome Page Recruitment AnalysisPlanning Design
  • 30. Initial Questions - Tips Recruitment AnalysisPlanning Design You may want to collect what type of experience users have with the product. You can use branching logic to give certain tasks to specific participants based on their expertise Avoid asking hypothetical questions. Participants are not good at knowing how they’ll behave in a certain situation. “Would you book flights on this website if we added this option?” is a bad example Be specific. “Daily” and “Weekly” is more specific than “Always” and “Sometimes”. Avoid “double questions” that have two questions in one. “Please indicate how easy and fast it was for you to complete the task?” Avoid asking yes/no questions. It can lead participants into an extreme statement. “Do you use this website all the time?” is not a good example. Don’t reveal too much information to the participant upfront Brand recognition can have a large impact on people’s reactions to products.
  • 31. Initial Question Examples Recruitment AnalysisPlanning Design
  • 32. Initial Question – Logic & Conditions Recruitment AnalysisPlanning Design • Skip participants to a specific task or question • Hide or show questions based on answers (example above) • Disqualify participants
  • 33. Selecting the Right Tasks Recruitment AnalysisPlanning Design Balance of User Data and Business Goals • What are users coming to your site to do (core tasks) ? • Are there any tasks that are required to use your site? • What drives revenue on your site?
  • 34. Selecting the Right Tasks Recruitment AnalysisPlanning Design Balance of User Data and Business Goals • What are users coming to your site to do (core tasks) ? - Find and watch a TV show • Are there any tasks that are required to use your site? - Create an account? • What drives revenue on your site? - Sign up for a premium account
  • 35. Task Construction - Tips Recruitment AnalysisPlanning Design Tasks should be natural and easy to understand, but not easy to guess Randomize task order to minimize the learning effect, if tasks are not dependent on each other. Create tasks that are easy to validate (you need to be able to confirm that the participant actually found an answer). Tasks should be worded in a way that doesn’t lead the participant When using Validation word tasks so that there can be a discrete answer (price, date etc.) Include “none of the above” option (it will discourage participants from using deduction to figure out the answer).
  • 36. Building Multiple Tasks - Randomizing Order Recruitment AnalysisPlanning Design Screener Pre-task questions Hulu Task 1 Hulu Task 2 Control for learning effects • Participants become more proficient as they use the site There are exceptions, sometimes tasks must be completed in order Final QuestionsRandomized
  • 37. Task Validation (3 options) Recruitment AnalysisPlanning Design • Validate by question • “Quiz question” used to verify task completion • Validate by URL • Successful if participant reach a specific URL • Think carefully about task wording and defining success • “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search results page vs. the specific episode page) • Self-reported
  • 38. Task Description Example Recruitment AnalysisPlanning Design
  • 39. Post Task Questions Recruitment AnalysisPlanning Design Self-Reported Data • Are user satisfied with the experience? • Do users believe they completed the task successfully? • Why wasn’t a participant successful ? Task Success Button Validation Abandon Button Abandon Questions Error Questions Success Questions
  • 40. Post Task Question Examples Recruitment AnalysisPlanning Design Validation Question Error Question Success
  • 41. Final Questionnaire - Tips Recruitment AnalysisPlanning Design Overall, how easy or difficult was it to use this website? How likely is it that you would recommend this website to a friend or colleague? After your experience with the website today, which of the following words do you associate with the brand? When using Validation word tasks so that there can be a discrete answer (price, date etc.) OVERALL RATING SCALES NET PROMOTER SCORE CHANGE IN BRAND PERCEPTION OPEN-ENDED QUESTIONS
  • 42. Final Questionnaire Example Questions Recruitment AnalysisPlanning Design
  • 43. What about recruiting participants? AnalysisPlanning Design Recruitment Define a screener Hulu Demo Study • Representative users • Demographics (gender, age etc.) • Control for experience • Larger sample size -> more precise predictions • Set profiles and define quota limits • Set quality controls
  • 44. What about recruiting participants? AnalysisPlanning Design Recruitment
  • 45. Quality Control Settings AnalysisPlanning Design Recruitment Quality Controls • Be careful not to exclude fast users • Cheaters often take the most efficient path (look out for “0 clicks”)
  • 46. What about recruiting participants?  3 choices to recruit participants: 1. Panel vendor (SSI, Research Now, etc) 2. Private Mailing List 3. Intercept visitors AnalysisPlanning Design Recruitment
  • 47. Hulu Demo Walkthrough Planning Design Recruitment Analysis
  • 48. Analyzing the Results Planning Design Recruitment Analysis 1. Task Success 2. Efficiency (time, clicks) 3. Self-Reported Satisfaction 4. Open prompts 5. Behavioral Data
  • 49. Analyzing the Results – Clean Data Planning Design Recruitment Analysis Review your data for outliers / “bad data” • Very High / Very Low task times • 0 Clicks • Nonsense Answers
  • 50. Make assumptions about user population (Confidence Intervals) Hulu baselines can be compared with data from future study (Significance Testing) Analyzing Metrics – Beyond the Tool Planning Design Recruitment Analysis Jeff Sauro, StatUsabilityPak
  • 51. Analyzing Metrics – Open Ended Planning Design Recruitment Analysis Use Text Clouds & Search for Patterns Use Comment Coding for more Detailed Analysis
  • 52. Analyzing Metrics – Behavioral Planning Design Recruitment Analysis What strategies are being used (how successful are they?) What are “Non-Success” participants doing? (filter data)
  • 53. Q&A
  • 54. Thanks so much for your time! Check out our upcoming news and events at www.userzoom.com Sunnyvale (USA) 440 N. Wolfe Rd. Sunnyvale, CA 94085 Phone: +1 (408) 524 7445 Contact: Alfonso de la Nuez Barcelona (Spain) Av. Diagonal 419 3º 2ª 08008 Barcelona Phone: +34 93 414 7554 Fax: +34 93 209 8380 Contact: Xavier Mestres Cheshire (UK) Booths Park. Chelford Road, Knutsford, Cheshire WA16 8GS Phone: + 44 (0)1565 759890 Cell: + 44 (0)7900 472 920 Contact: Arthur Moan