• Share
  • Email
  • Embed
  • Like
  • Private Content
Webinar: How to Conduct Unmoderated Remote Usability Testing
 

Webinar: How to Conduct Unmoderated Remote Usability Testing

on

  • 1,557 views

 

Statistics

Views

Total Views
1,557
Views on SlideShare
1,178
Embed Views
379

Actions

Likes
5
Downloads
52
Comments
0

2 Embeds 379

http://info.userzoom.com 351
https://na-b.marketodesigner.com 28

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Saves time – Very fast, thousands on panels, Money – essence of quick and dirty. Techniques for dealing with noise, unrealistic to be in the lab that long. Combines both qual/quant and attitudes and behavior.
  • Thanks going to cover the process of creating a study and the steps involvedBraking down in 4 topic areas
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesIllustrate again very large space. We’re just going to talk about this
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesIllustrate again very large space. We’re just going to talk about this
  • Unmoderated remote research is a massive space. We need to focus on a specific goalHere is a list (not comprehensive)-We can group these with methodologiesEvaluate and Quantify UX  Baseline study we can compare results as we redesign or over timeTask-Based users going to be navigation on websiteIllustrate again very large space. We’re just going to talk about this
  • Disclaimer : not a customer just a demo example
  • Our Goal  Evaluation and Quantifying the UX Baseline Test we can compare our results laterPartial list from measuring usability (there are several of these)Completion Rate  providing tasks to users, if they can’t complete the tasks not much else matters, going to be a primary focus.We also have a high sample size. Important for analysis. we’re going to be able to make better predictions than what you would get in a lab study with 3-5 users.We’re also going to be looking at efficiency(task time), satisfaction. Triangulate data. Tell a more powerful storyBehavioral data too. Where did people go and what did they click Plan on what metrics you want and know how you can use them.
  • Lets get started We need to have a plan  here is a simple outlineWe have a welcome pageInitial questionsTasks where users are going to be sent to the Hulu websiteQuestions specific to those tasksFinal questionsGonna walkthrough each of these areas
  • Set Expectations
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Bad/Good Examples
  • Gather feedback on the holistic experiencePre-post metrics
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • Disclaimer : not a customer just a demo example
  • Success are people able to completeEfficiency not as crucial for Hulu depends on your domain. We can identify problem areas
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population
  • External Validity Trying to understand the experience of our entire user population

Webinar: How to Conduct Unmoderated Remote Usability Testing Webinar: How to Conduct Unmoderated Remote Usability Testing Presentation Transcript

  • Webinar:How to Conduct Unmoderated RemoteUsability TestingMay 30th, 2012
  • Intros Alfonso de la Nuez John Romadka Nate ColkerCo-Founder & Co-CEO UX Researcher UX Researcher UserZoom Optum Insight UserZoom (Division of UnitedHealth Group) Twitter Hashtag #uzwebinar
  • OverviewPart 1 (20 min) Introduction to unmoderated remote usability testing A real life case study featuring a company in the insurance industryPart 2 (30 min) How to plan for an unmoderated remote usability study How to design a study What metrics you need to collect How to recruit participants How to analyze the data Q&A
  • User testing, feedback & analytics tools mania!
  • Let’s try to make some sense out of this…Many ways of organizing remote testing tools: Moderated vs. Unmoderated Qualitative vs. Quantitative-based User engaging vs. Non-user engaging (analytics) Usability focused vs. ‘quick user feedback’ (VOC) Video vs. Stats/metrics Etc.
  • Remote, unmoderated - A growing research method
  • Why this testing tool fever? Consumers more empowered than ever, thx in part to Social Media Ecommerce maturing, more competition Web analytics, usability test in the lab, mkt research… no longer enough or are very costly Agile dev is all about speed, iteration 1. Designing & Testing UX quickly & cost-effectively is KEY 2. People want better!
  • Why Should You Care about Remote Testing? Cost-effective, save time and money on research You can quantify usability, obtain statistically significant UX data Run competitive UX Benchmarking Iterative testing, validate design/prototypes Test with users from different geographic locations Objectivity + accountability Improves communication Mitigate risks
  • What can be tested with URUT?  Static mockups  Wireframes  Prototypes  Multimedia  Live websites  Intranets  …any web-based UI  …anything you can host on a server and access through a web browser
  • Pros & ConsPros: Cost-effectiveness Quantify UX Geographic representation Honest feedback Combination of results and metricsCons: Obviously… no face to face
  • Case Study:Remote UX Studies - Tools & MethodsJohn Romadka, UX ResearcherOptumInsight (Division of UnitedHealth Group)
  • ChallengesNew projectWe’d depleted our pool of internal (agent) participantslocallyNeeded to get a broader perspective from agentsacross countryNetwork security issuesLimited number of dedicated UX Researchers
  • ChallengesCouldn’t convince business partners to test iterativelyLab studies were costlyand often resulted inlarge margin of error 90% confidence interval
  • Making the CaseMade the unmoderated testing business case More cost efficient • Unmoderated testing can cost 50% less than traditional moderated testing • Primarily due to reduction in effort hours spent in lab Timeline • Reduced the 3 week lead time for recruiting to less than 1 week.
  • Making the CaseValidation Testing Retest within days, instead of week Moderated test: 180 hours vs. Unmoderated test: 40 hours Remote Unmoderated Testing performed for ~20% the cost of benchmark Moderated test Test 1- (Moderated) = $10,000 Test 2- (Unmoderated) = $2000)Improved Confidence Intervals
  • ResultsSuccessful company-wide adoption of UserZoomObtained a pilot group of “Field” participants Broad range of demographics, locationsBi-weekly expectation of participating in whatever UXresearch was available and ready to study • Required less context, setup
  • ResultsUserZoom allowed us to quickly setup/run/analyze Online Surveys Task-based studies Card Sort exercises Tree Testing studies Task-based studies: Moderated Remote Unmoderated Remote Moderated Lab
  • ResultsTask Based Studies -Remote Moderated: combine 3 tools UserZoom (metrics) Lead users through study Central point for metrics Quick and easy to re-test Webex (screen sharing, audio) Deployed company-wide Giving participant control of web app on dev Morae (note taking, recording)
  • ResultsTask Based Studies -Combined Unmoderated/Moderated Users were onsite occasionally Validate our unmoderated resultsUserZoom: Used same setup for both Consistent test method and process Freed up the Facilitator to observe, take notes. Knowing UserZoom was gathering metrics
  • Thank You! Gplus.to/jromadka @jaromad
  • How to Conduct Unmoderated Remote Usability Testing• How to plan for an unmoderated remote usability study• What metrics you need to collect• How to design a study• How to recruit participants• How to analyze the data Planning Design Recruitment Analysis
  • What are the goals of the study?A Few Examples….Evaluate and Quantify the User ExperienceComparing Multiple Designs (A/B)Competitive BenchmarkIdentify Visitors’ Core TasksRedesign Navigation StructureImprove the Navigation Experience Planning Design Recruitment Analysis
  • What are the goals of the study?A Few Examples…. MethodEvaluate and Quantify the User ExperienceComparing Multiple Designs (A/B) Remote Task-BasedCompetitive BenchmarkIdentify Visitors’ Core Tasks True IntentRedesign Navigation Structure Card Sort / Tree TestImprove the Navigation Experience Planning Design Recruitment Analysis
  • What are the goals of the study?A Few Examples…. MethodEvaluate and Quantify the User ExperienceComparing Multiple Designs (A/B) Remote Task-BasedCompetitive BenchmarkIdentify Visitors’ Core Tasks True IntentRedesign Navigation Structure Card Sort / Tree TestImprove the Navigation Experience Planning Design Recruitment Analysis
  • A Hulu Demo Study Planning Design Recruitment Analysis
  • What Metrics do you Need to Collect?  Jeff Sauro, Measuringusability.com Planning Design Recruitment Analysis
  • Create A Study Script • Study Goals & Participant Profile • Welcome Page • Initial Questions • Task 1 • Post-Task Questions • Task 2 • Post-Task Questions • Final Questionnaire Planning Design Recruitment Analysis
  • Welcome Page Design Technical Time requirements estimate Purpose of Incentive Instructions the studySet Expectations for the participants• Introduce study• Time estimate and incentive• Keep short and to the point Planning Design Recruitment Analysis
  • Example Welcome Page Planning Design Recruitment Analysis
  • Initial Questions - TipsYou may want to Avoid asking hypothetical Be specific. “Daily”collect what type of questions. Participants and “Weekly” isexperience users are not good at knowing more specific thanhave with the product. how they’ll behave in a “Always” andYou can use branching logic to certain situation. “Sometimes”.give certain tasks to specific “Would you book flights onparticipants based on their this website if we added thisexpertise option?” is a bad exampleAvoid “double Avoid asking yes/no Don’t reveal tooquestions” that have questions. It can lead much informationtwo questions in one. participants into an to the participant extreme statement. upfront“Please indicate how easyand fast it was for you to “Do you use this website all Brand recognition cancomplete the task?” the time?” is not a good have a large impact on example. people’s reactions to products. Planning Design Recruitment Analysis
  • Initial Question Examples Planning Design Recruitment Analysis
  • Initial Question – Logic & Conditions• Skip participants to a specific task or question• Hide or show questions based on answers (example above)• Disqualify participants Planning Design Recruitment Analysis
  • Selecting the Right TasksBalance of User Data and Business Goals• What are users coming to your site to do (core tasks) ?• Are there any tasks that are required to use your site?• What drives revenue on your site? Planning Design Recruitment Analysis
  • Selecting the Right TasksBalance of User Data and Business Goals• What are users coming to your site to do (core tasks) ? - Find and watch a TV show• Are there any tasks that are required to use your site? - Create an account?• What drives revenue on your site? - Sign up for a premium account Planning Design Recruitment Analysis
  • Task Construction - TipsTasks should be Randomize task order Create tasks thatnatural and easy to to minimize the are easy tounderstand, but not learning effect, if tasks validate (you needeasy to guess are not dependent on to be able to each other. confirm that the participant actually found an answer).Tasks should be When using Validation Include “none ofworded in a way that word tasks so that the above” optiondoesn’t lead the there can be a (it will discourageparticipant discrete answer participants from (price, date etc.) using deduction to figure out the answer). Planning Design Recruitment Analysis
  • Building Multiple Tasks - Randomizing Order Hulu Task 1 Screener Pre-task questions Randomized Final Questions Hulu Task 2Control for learning effects• Participants become more proficient as they use the siteThere are exceptions, sometimes tasks must be completed in order Planning Design Recruitment Analysis
  • Task Validation (3 options)• Validate by question • “Quiz question” used to verify task completion• Validate by URL • Successful if participant reach a specific URL • Think carefully about task wording and defining success • “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search results page vs. the specific episode page)• Self-reported Planning Design Recruitment Analysis
  • Task Description Example Planning Design Recruitment Analysis
  • Post Task Questions Success Questions Success Task Button Validation Error Questions Abandon Button Abandon QuestionsSelf-Reported Data• Are user satisfied with the experience?• Do users believe they completed the task successfully?• Why wasn’t a participant successful ? Planning Design Recruitment Analysis
  • Post Task Question Examples Success Validation Question Error Question Planning Design Recruitment Analysis
  • Final Questionnaire - Tips OVERALL RATING NET PROMOTER SCALES SCORE How likely is it that Overall, how easy or you would recommend difficult was it to use this website to a friend this website? or colleague? CHANGE IN BRAND OPEN-ENDED QUESTIONS PERCEPTION When using Validation After your experience word tasks so that with the website there can be a today, which of the discrete answer following words do (price, date etc.) you associate with the brand? Planning Design Recruitment Analysis
  • Final Questionnaire Example Questions Planning Design Recruitment Analysis
  • What about recruiting participants?Define a screenerHulu Demo Study• Representative users • Demographics (gender, age etc.)• Control for experience• Larger sample size -> more precise predictions• Set profiles and define quota limits• Set quality controls Planning Design Recruitment Analysis
  • What about recruiting participants? Planning Design Recruitment Analysis
  • Quality Control SettingsQuality Controls• Be careful not to exclude fast users• Cheaters often take the most efficient path (look out for “0 clicks”) Planning Design Recruitment Analysis
  • What about recruiting participants? 3 choices to recruit participants: 1. Panel vendor (SSI, Research Now, etc) 2. Private Mailing List 3. Intercept visitors Planning Design Recruitment Analysis
  • Hulu Demo Walkthrough Planning Design Recruitment Analysis
  • Analyzing the Results 1. Task Success 2. Efficiency (time, clicks) 3. Self-Reported Satisfaction 4. Open prompts 5. Behavioral Data Planning Design Recruitment Analysis
  • Analyzing the Results – Clean DataReview your data for outliers / “bad data”• Very High / Very Low task times• 0 Clicks• Nonsense Answers Planning Design Recruitment Analysis
  • Analyzing Metrics – Beyond the ToolMake assumptions about user population (Confidence Intervals) Jeff Sauro, StatUsabilityPakHulu baselines can be compared with data from future study (Significance Testing) Planning Design Recruitment Analysis
  • Analyzing Metrics – Open EndedUse Text Clouds & Search for PatternsUse Comment Coding for more Detailed Analysis Planning Design Recruitment Analysis
  • Analyzing Metrics – BehavioralWhat strategies are being used (how successful are they?)What are “Non-Success” participants doing? (filter data) Planning Design Recruitment Analysis
  • Q&A
  • Thanks so much for your time! Check out our upcoming news and events at www.userzoom.comSunnyvale (USA) Barcelona (Spain) Cheshire (UK)440 N. Wolfe Rd. Sunnyvale, Av. Diagonal 419 3º 2ª 08008 Booths Park. Chelford Road, Knutsford,CA 94085 Barcelona Phone: +34 93 414 Cheshire WA16 8GSPhone: +1 (408) 524 7445 7554 Fax: +34 93 209 8380 Phone: + 44 (0)1565 759890Contact: Alfonso de la Nuez Contact: Xavier Mestres Cell: + 44 (0)7900 472 920 Contact: Arthur Moan