Webinar: How to Conduct Unmoderated Remote Usability Testing

Webinar:How to Conduct Unmoderated RemoteUsability TestingMay 30th, 2012

Intros Alfonso de la Nuez John Romadka Nate ColkerCo-Founder & Co-CEO UX Researcher UX Researcher UserZoom Optum Insight UserZoom (Division of UnitedHealth Group) Twitter Hashtag #uzwebinar

OverviewPart 1 (20 min) Introduction to unmoderated remote usability testing A real life case study featuring a company in the insurance industryPart 2 (30 min) How to plan for an unmoderated remote usability study How to design a study What metrics you need to collect How to recruit participants How to analyze the data Q&A

User testing, feedback & analytics tools mania!

Let’s try to make some sense out of this…Many ways of organizing remote testing tools: Moderated vs. Unmoderated Qualitative vs. Quantitative-based User engaging vs. Non-user engaging (analytics) Usability focused vs. ‘quick user feedback’ (VOC) Video vs. Stats/metrics Etc.

Remote, unmoderated - A growing research method

Why this testing tool fever? Consumers more empowered than ever, thx in part to Social Media Ecommerce maturing, more competition Web analytics, usability test in the lab, mkt research… no longer enough or are very costly Agile dev is all about speed, iteration 1. Designing & Testing UX quickly & cost-effectively is KEY 2. People want better!

Why Should You Care about Remote Testing? Cost-effective, save time and money on research You can quantify usability, obtain statistically significant UX data Run competitive UX Benchmarking Iterative testing, validate design/prototypes Test with users from different geographic locations Objectivity + accountability Improves communication Mitigate risks

What can be tested with URUT?  Static mockups  Wireframes  Prototypes  Multimedia  Live websites  Intranets  …any web-based UI  …anything you can host on a server and access through a web browser

Pros & ConsPros: Cost-effectiveness Quantify UX Geographic representation Honest feedback Combination of results and metricsCons: Obviously… no face to face

Case Study:Remote UX Studies - Tools & MethodsJohn Romadka, UX ResearcherOptumInsight (Division of UnitedHealth Group)

ChallengesNew projectWe’d depleted our pool of internal (agent) participantslocallyNeeded to get a broader perspective from agentsacross countryNetwork security issuesLimited number of dedicated UX Researchers

ChallengesCouldn’t convince business partners to test iterativelyLab studies were costlyand often resulted inlarge margin of error 90% confidence interval

Making the CaseMade the unmoderated testing business case More cost efficient • Unmoderated testing can cost 50% less than traditional moderated testing • Primarily due to reduction in effort hours spent in lab Timeline • Reduced the 3 week lead time for recruiting to less than 1 week.

Making the CaseValidation Testing Retest within days, instead of week Moderated test: 180 hours vs. Unmoderated test: 40 hours Remote Unmoderated Testing performed for ~20% the cost of benchmark Moderated test Test 1- (Moderated) = $10,000 Test 2- (Unmoderated) = $2000)Improved Confidence Intervals

ResultsSuccessful company-wide adoption of UserZoomObtained a pilot group of “Field” participants Broad range of demographics, locationsBi-weekly expectation of participating in whatever UXresearch was available and ready to study • Required less context, setup

ResultsUserZoom allowed us to quickly setup/run/analyze Online Surveys Task-based studies Card Sort exercises Tree Testing studies Task-based studies: Moderated Remote Unmoderated Remote Moderated Lab

ResultsTask Based Studies -Remote Moderated: combine 3 tools UserZoom (metrics) Lead users through study Central point for metrics Quick and easy to re-test Webex (screen sharing, audio) Deployed company-wide Giving participant control of web app on dev Morae (note taking, recording)

ResultsTask Based Studies -Combined Unmoderated/Moderated Users were onsite occasionally Validate our unmoderated resultsUserZoom: Used same setup for both Consistent test method and process Freed up the Facilitator to observe, take notes. Knowing UserZoom was gathering metrics

Thank You! Gplus.to/jromadka @jaromad

How to Conduct Unmoderated Remote Usability Testing• How to plan for an unmoderated remote usability study• What metrics you need to collect• How to design a study• How to recruit participants• How to analyze the data Planning Design Recruitment Analysis

What are the goals of the study?A Few Examples….Evaluate and Quantify the User ExperienceComparing Multiple Designs (A/B)Competitive BenchmarkIdentify Visitors’ Core TasksRedesign Navigation StructureImprove the Navigation Experience Planning Design Recruitment Analysis

What are the goals of the study?A Few Examples…. MethodEvaluate and Quantify the User ExperienceComparing Multiple Designs (A/B) Remote Task-BasedCompetitive BenchmarkIdentify Visitors’ Core Tasks True IntentRedesign Navigation Structure Card Sort / Tree TestImprove the Navigation Experience Planning Design Recruitment Analysis

A Hulu Demo Study Planning Design Recruitment Analysis

What Metrics do you Need to Collect?  Jeff Sauro, Measuringusability.com Planning Design Recruitment Analysis

Create A Study Script • Study Goals & Participant Profile • Welcome Page • Initial Questions • Task 1 • Post-Task Questions • Task 2 • Post-Task Questions • Final Questionnaire Planning Design Recruitment Analysis

Welcome Page Design Technical Time requirements estimate Purpose of Incentive Instructions the studySet Expectations for the participants• Introduce study• Time estimate and incentive• Keep short and to the point Planning Design Recruitment Analysis

Example Welcome Page Planning Design Recruitment Analysis

Initial Questions - TipsYou may want to Avoid asking hypothetical Be specific. “Daily”collect what type of questions. Participants and “Weekly” isexperience users are not good at knowing more specific thanhave with the product. how they’ll behave in a “Always” andYou can use branching logic to certain situation. “Sometimes”.give certain tasks to specific “Would you book flights onparticipants based on their this website if we added thisexpertise option?” is a bad exampleAvoid “double Avoid asking yes/no Don’t reveal tooquestions” that have questions. It can lead much informationtwo questions in one. participants into an to the participant extreme statement. upfront“Please indicate how easyand fast it was for you to “Do you use this website all Brand recognition cancomplete the task?” the time?” is not a good have a large impact on example. people’s reactions to products. Planning Design Recruitment Analysis

Initial Question Examples Planning Design Recruitment Analysis

Initial Question – Logic & Conditions• Skip participants to a specific task or question• Hide or show questions based on answers (example above)• Disqualify participants Planning Design Recruitment Analysis

Selecting the Right TasksBalance of User Data and Business Goals• What are users coming to your site to do (core tasks) ?• Are there any tasks that are required to use your site?• What drives revenue on your site? Planning Design Recruitment Analysis

Selecting the Right TasksBalance of User Data and Business Goals• What are users coming to your site to do (core tasks) ? - Find and watch a TV show• Are there any tasks that are required to use your site? - Create an account?• What drives revenue on your site? - Sign up for a premium account Planning Design Recruitment Analysis

Task Construction - TipsTasks should be Randomize task order Create tasks thatnatural and easy to to minimize the are easy tounderstand, but not learning effect, if tasks validate (you needeasy to guess are not dependent on to be able to each other. confirm that the participant actually found an answer).Tasks should be When using Validation Include “none ofworded in a way that word tasks so that the above” optiondoesn’t lead the there can be a (it will discourageparticipant discrete answer participants from (price, date etc.) using deduction to figure out the answer). Planning Design Recruitment Analysis

Building Multiple Tasks - Randomizing Order Hulu Task 1 Screener Pre-task questions Randomized Final Questions Hulu Task 2Control for learning effects• Participants become more proficient as they use the siteThere are exceptions, sometimes tasks must be completed in order Planning Design Recruitment Analysis

Task Validation (3 options)• Validate by question • “Quiz question” used to verify task completion• Validate by URL • Successful if participant reach a specific URL • Think carefully about task wording and defining success • “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search results page vs. the specific episode page)• Self-reported Planning Design Recruitment Analysis

Task Description Example Planning Design Recruitment Analysis

Post Task Questions Success Questions Success Task Button Validation Error Questions Abandon Button Abandon QuestionsSelf-Reported Data• Are user satisfied with the experience?• Do users believe they completed the task successfully?• Why wasn’t a participant successful ? Planning Design Recruitment Analysis

Post Task Question Examples Success Validation Question Error Question Planning Design Recruitment Analysis

Final Questionnaire - Tips OVERALL RATING NET PROMOTER SCALES SCORE How likely is it that Overall, how easy or you would recommend difficult was it to use this website to a friend this website? or colleague? CHANGE IN BRAND OPEN-ENDED QUESTIONS PERCEPTION When using Validation After your experience word tasks so that with the website there can be a today, which of the discrete answer following words do (price, date etc.) you associate with the brand? Planning Design Recruitment Analysis

Final Questionnaire Example Questions Planning Design Recruitment Analysis

What about recruiting participants?Define a screenerHulu Demo Study• Representative users • Demographics (gender, age etc.)• Control for experience• Larger sample size -> more precise predictions• Set profiles and define quota limits• Set quality controls Planning Design Recruitment Analysis

What about recruiting participants? Planning Design Recruitment Analysis

Quality Control SettingsQuality Controls• Be careful not to exclude fast users• Cheaters often take the most efficient path (look out for “0 clicks”) Planning Design Recruitment Analysis

What about recruiting participants? 3 choices to recruit participants: 1. Panel vendor (SSI, Research Now, etc) 2. Private Mailing List 3. Intercept visitors Planning Design Recruitment Analysis

Hulu Demo Walkthrough Planning Design Recruitment Analysis

Analyzing the Results 1. Task Success 2. Efficiency (time, clicks) 3. Self-Reported Satisfaction 4. Open prompts 5. Behavioral Data Planning Design Recruitment Analysis

Analyzing the Results – Clean DataReview your data for outliers / “bad data”• Very High / Very Low task times• 0 Clicks• Nonsense Answers Planning Design Recruitment Analysis

Analyzing Metrics – Beyond the ToolMake assumptions about user population (Confidence Intervals) Jeff Sauro, StatUsabilityPakHulu baselines can be compared with data from future study (Significance Testing) Planning Design Recruitment Analysis

Analyzing Metrics – Open EndedUse Text Clouds & Search for PatternsUse Comment Coding for more Detailed Analysis Planning Design Recruitment Analysis

Analyzing Metrics – BehavioralWhat strategies are being used (how successful are they?)What are “Non-Success” participants doing? (filter data) Planning Design Recruitment Analysis

Thanks so much for your time! Check out our upcoming news and events at www.userzoom.comSunnyvale (USA) Barcelona (Spain) Cheshire (UK)440 N. Wolfe Rd. Sunnyvale, Av. Diagonal 419 3º 2ª 08008 Booths Park. Chelford Road, Knutsford,CA 94085 Barcelona Phone: +34 93 414 Cheshire WA16 8GSPhone: +1 (408) 524 7445 7554 Fax: +34 93 209 8380 Phone: + 44 (0)1565 759890Contact: Alfonso de la Nuez Contact: Xavier Mestres Cell: + 44 (0)7900 472 920 Contact: Arthur Moan

http://info.userzoom.com	239
https://na-b.marketodesigner.com	28

Webinar: How to Conduct Unmoderated Remote Usability Testing

by UserZoom on May 31, 2012

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 267

Statistics

Webinar: How to Conduct Unmoderated Remote Usability Testing Presentation Transcript