Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Best practices for remote usability testing


Published on

Slides from the webinar on Best Practices for Remote User Testing, led by a real guru on measuring UX, Jeff Sauro. Jeff presents best practices and a real case study that shows how to use UserZoom and …

Slides from the webinar on Best Practices for Remote User Testing, led by a real guru on measuring UX, Jeff Sauro. Jeff presents best practices and a real case study that shows how to use UserZoom and UserTesting simultaneously for capturing quantitative and qualitative data.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Parallel testingSolo UserZoomSolo Usertesting
  • SUM is the average of task completion rate, task-level satisfaction and the standardized task-time (from completed users only).You can interpret these numbers by saying “Renting a car on Budget has a usability score of 58% out of 100%”
  • Budget_Good_Enterprise Bad
  • Need 1 Users to have an 85% chance of detecting p = .80Need 56 Users for p = .033
  • Budget_Portland
  • Transcript

    • 1. Best Practices for Remote UsabilityTestingComparative Benchmark ResultsJeff Sauro | Measuring Usability LLC2/28/2012
    • 2. 5 Challenges Facing UX Professionals Today1. Time2. Cost3. Tools & Techniques4. Representative Users5. Deliverables Measuring Usability LLC
    • 3. Measuring Usability LLC
    • 4. DeliverablesBusiness Requirement Doc Personas WireframesFunction Requirement Spec Usability Test Plan User ScreenerTask Analysis Doc Recruitment Doc Usability Report Measuring Usability LLC
    • 5. Representative Measuring Usability LLC
    • 6. Unmoderated Remote Testing w/ Video: Not aPanacea but pretty close Measuring Usability LLC
    • 7. Remote Testing Setup using UserZoom & Usertesting Website is shown in main window Users are shown the task and asked to complete it. Full A/V available online.Collects Task Time, Completion Rates, Task-Ease and Click Paths Measuring Usability LLC
    • 8. Case Study: Measuring Usability LLC
    • 9. Summary• 61 Total Remote Unmoderated Users• 4 Tasks• Within Subjects• Randomized Tasks & Websites• Recruitment Criteria: > 25 and US Based – 40 % Had Rented a Car from Budget or Enterprise• Mix of Age, Geography• 58% Female• Total Study Length : 20-30 minutes• Users Recruited and Data collected in 48 Hours Measuring Usability LLC
    • 10. 2 Tasks1. Find the Nearest LocationFind the address of the nearest rental office to the Hilton Hotel located at 921 SW Sixth Avenue, Portland, Oregon, United States 97204. Write down or copy the name of the street of the nearest location.2. Renting a CarLocation: Logan Airport, Boston MARental Period : Fri April, 13th 2012 at 11.00am to Sunday April 15th at 5.00pm.Class: Intermediate (NO SUV’s)Extras: GPS Navigation and Car Seat for a 2 Year oldIf asked, use the following identification: Name : John Smith Email: Phone : 303-555-1212 Credit Card: Visa Measuring Usability LLC
    • 11. Task Completion RatesError Bars are 90% Confidence Intervals* p <. 01 ; N = 62 Measuring Usability LLC
    • 12. Reasons for Failure by Task Measuring Usability LLC
    • 13. Disaster Rates (Highest Confidence but Failure) Measuring Usability LLC
    • 14. Task-Level Ease : Single Ease Question (SEQ)Very EasyVery Hard Error Bars are 90% Confidence Intervals * p < . 01 ; N = 62 Measuring Usability LLC
    • 15. Task Time: Completed Tasks OnlySeconds # Users Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
    • 16. Single Usability Metric (SUM)Avg Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
    • 17. Completed Times: Find Nearest Location Median Time with all Users: 119 Median Time Removed Long Times: 118 Measuring Usability LLC
    • 18. Why did the user take 7 minutes? - Survey taker Measuring Usability LLC
    • 19. Why? Measuring Usability LLC
    • 20. Unmoderated Testing with Video Measuring Usability LLC
    • 21. VerbatimsTotalingIt didn’t give the whole total with the extras-I had to add that myself.Took the price of the car and manually added on 3 day price of car seat ($10.99*3) and same for GPS but still unsure of total because unsure if these items are taxed.I could not get a total price of the rental reservation, and the price I got was not a possible answer on the survey.It told me that the price did not include the add-ons, and that those *may* be charged at the counter. I don’t know what it means by may be charged at the counter- does that mean they many not be charged for? I don’t understand why it doesn’t give a total cost of the rental with the GPS and toddler seat included. I added them in myself with a calculator, but I’m thought it would just automatically show me what the total will be.I was unable to find the link to add the GPS. I did read that it is 10.99 per day. I was also not able to find the link to add the car didn’t give an exact total for the booking. you said that the additional charges would be done at the counter. i think you should have gave me an estimated total so i wouldn’t have had to compute the charges myself.I just really wasn’t sure of my final pricing or even if I booked it right.I had to use a calculator to add the GPS and car seat charges to the car rental. I would have preferred if I could have selected the add-ons, and then it would show the total revised price before I booked the reservation. I was wondering if the add-ons would also have other charges or taxes. overall easier than the last one.Hard to know if updated price when adding the GPS and car seat"The website was confusing to me. I didn’t understand why the rental amount did not display an increase after I added the GPS and child seat to my order - it seemed like something was working incorrectly. I even accidentally submitted the reservation, as I was trying to find that total.Basically, the website was not user-friendly. Items were difficult to find, and the interface was confusing."It was hard to understand what the total cost would be with the add-ons included.Finding the ExtrasI saw no area to add the extrasI saw no way to add GPS or child seat.Because I didn’t know at what point it would offer the add-ons, and the total didn’t include the gps and car seat.It wasn’t clear where to add the extras and once they were added it wasn’t clear what the final total would be. I also thought payment details needed to be added before an actual booking would take place and they weren’t.I couldn’t find the address for either task. I couldn’t figure out the options until I noticed them after I chose the car.. Measuring Usability LLC
    • 22. Reasons for Low Task Ratings on Renting a Car Error Bars are 90% Confidence Intervals Measuring Usability LLC
    • 23. Total Price Problem C0xflQ%3d%3d Measuring Usability LLC
    • 24. Finding Location & Search Results Measuring Usability LLC
    • 25. Usability Problems: Renting a Car Extras Total Search ProblemUser Measuring Usability LLC
    • 26. User by Problem Matrices Rent a Car Find a Location ProblemUser P = .12 P = .15 P = .09 P = .07 Measuring Usability LLC
    • 27. Long Tail Problems file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d Measuring Usability LLC
    • 28. Default Location Measuring Usability LLC
    • 29. 1st Click Analysis Measuring Usability LLC
    • 30. First Click Analysis • If a user’s first click is down the wrong-path, only 46% eventually succeed • If a user’s first click is down the right-path, 87% eventually succeedWolfson, et al (2008) Contextual card sorting (or FirstClick testing): A new methodology forvalidating information architectures, UPA Proceedings Measuring Usability LLC
    • 31. Find the Nearest Location: 83% Complete42% Of Users84% Success50% Of Users80% Success 3% Of Users 100% Success Measuring Usability LLC
    • 32. Find the Nearest Location: 71% Complete8% Of Users100% Success40% Of Users 10% Of Users71% Success 83% Success10% Of Users50% Success Measuring Usability LLC
    • 33. Test Level Metrics Measuring Usability LLC
    • 34. Attitude before and After Usability Test VeryFavorableUn-FavorableVery Diff % After users completed the tasks Budget’s Score increased 12% & Enterprises decreased 15% * P < .05 ; Error Bars are 90% Confidence Intervals and Paired t-test Used for Statistical Comparison Measuring Usability LLC
    • 35. Net Promoter ScoreHow likely is it that you’ll recommend this website to a friend? Enterprise Budget 30th Percentile of Websites 73rd Percentile of Websites Measuring Usability LLC
    • 36. Measuring Usability LLC
    • 37. SUPR-Q and Sub-Scales Measuring Usability LLC
    • 38. Takeaways & Tips• Pretest Tasks and Study Design• Within Subjects• Compared to What?• Triangulate with Multiple Metrics• Use Standardized Questions – SEQ – SUPR-Q• Remote Unmoderated Testing w/ Video – In Parallel – As a Subset Measuring Usability LLC
    • 39. Find out More Sept 19-20th 2012 Hyatt Convention Center Denver, CO @LeanUXDenver Twitter @MsrUsability Measuring Usability LLC