Best practices for remote usability testing


Published on

Slides from the webinar on Best Practices for Remote User Testing, led by a real guru on measuring UX, Jeff Sauro. Jeff presents best practices and a real case study that shows how to use UserZoom and UserTesting simultaneously for capturing quantitative and qualitative data.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Parallel testingSolo UserZoomSolo Usertesting
  • SUM is the average of task completion rate, task-level satisfaction and the standardized task-time (from completed users only).You can interpret these numbers by saying “Renting a car on Budget has a usability score of 58% out of 100%”
  • Budget_Good_Enterprise Bad
  • Need 1 Users to have an 85% chance of detecting p = .80Need 56 Users for p = .033
  • Budget_Portland
  • Best practices for remote usability testing

    1. 1. Best Practices for Remote UsabilityTestingComparative Benchmark ResultsJeff Sauro | Measuring Usability LLC2/28/2012
    2. 2. 5 Challenges Facing UX Professionals Today1. Time2. Cost3. Tools & Techniques4. Representative Users5. Deliverables Measuring Usability LLC
    3. 3. Measuring Usability LLC
    4. 4. DeliverablesBusiness Requirement Doc Personas WireframesFunction Requirement Spec Usability Test Plan User ScreenerTask Analysis Doc Recruitment Doc Usability Report Measuring Usability LLC
    5. 5. Representative Measuring Usability LLC
    6. 6. Unmoderated Remote Testing w/ Video: Not aPanacea but pretty close Measuring Usability LLC
    7. 7. Remote Testing Setup using UserZoom & Usertesting Website is shown in main window Users are shown the task and asked to complete it. Full A/V available online.Collects Task Time, Completion Rates, Task-Ease and Click Paths Measuring Usability LLC
    8. 8. Case Study: Measuring Usability LLC
    9. 9. Summary• 61 Total Remote Unmoderated Users• 4 Tasks• Within Subjects• Randomized Tasks & Websites• Recruitment Criteria: > 25 and US Based – 40 % Had Rented a Car from Budget or Enterprise• Mix of Age, Geography• 58% Female• Total Study Length : 20-30 minutes• Users Recruited and Data collected in 48 Hours Measuring Usability LLC
    10. 10. 2 Tasks1. Find the Nearest LocationFind the address of the nearest rental office to the Hilton Hotel located at 921 SW Sixth Avenue, Portland, Oregon, United States 97204. Write down or copy the name of the street of the nearest location.2. Renting a CarLocation: Logan Airport, Boston MARental Period : Fri April, 13th 2012 at 11.00am to Sunday April 15th at 5.00pm.Class: Intermediate (NO SUV’s)Extras: GPS Navigation and Car Seat for a 2 Year oldIf asked, use the following identification: Name : John Smith Email: Phone : 303-555-1212 Credit Card: Visa Measuring Usability LLC
    11. 11. Task Completion RatesError Bars are 90% Confidence Intervals* p <. 01 ; N = 62 Measuring Usability LLC
    12. 12. Reasons for Failure by Task Measuring Usability LLC
    13. 13. Disaster Rates (Highest Confidence but Failure) Measuring Usability LLC
    14. 14. Task-Level Ease : Single Ease Question (SEQ)Very EasyVery Hard Error Bars are 90% Confidence Intervals * p < . 01 ; N = 62 Measuring Usability LLC
    15. 15. Task Time: Completed Tasks OnlySeconds # Users Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
    16. 16. Single Usability Metric (SUM)Avg Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
    17. 17. Completed Times: Find Nearest Location Median Time with all Users: 119 Median Time Removed Long Times: 118 Measuring Usability LLC
    18. 18. Why did the user take 7 minutes? - Survey taker Measuring Usability LLC
    19. 19. Why? Measuring Usability LLC
    20. 20. Unmoderated Testing with Video Measuring Usability LLC
    21. 21. VerbatimsTotalingIt didn’t give the whole total with the extras-I had to add that myself.Took the price of the car and manually added on 3 day price of car seat ($10.99*3) and same for GPS but still unsure of total because unsure if these items are taxed.I could not get a total price of the rental reservation, and the price I got was not a possible answer on the survey.It told me that the price did not include the add-ons, and that those *may* be charged at the counter. I don’t know what it means by may be charged at the counter- does that mean they many not be charged for? I don’t understand why it doesn’t give a total cost of the rental with the GPS and toddler seat included. I added them in myself with a calculator, but I’m thought it would just automatically show me what the total will be.I was unable to find the link to add the GPS. I did read that it is 10.99 per day. I was also not able to find the link to add the car didn’t give an exact total for the booking. you said that the additional charges would be done at the counter. i think you should have gave me an estimated total so i wouldn’t have had to compute the charges myself.I just really wasn’t sure of my final pricing or even if I booked it right.I had to use a calculator to add the GPS and car seat charges to the car rental. I would have preferred if I could have selected the add-ons, and then it would show the total revised price before I booked the reservation. I was wondering if the add-ons would also have other charges or taxes. overall easier than the last one.Hard to know if updated price when adding the GPS and car seat"The website was confusing to me. I didn’t understand why the rental amount did not display an increase after I added the GPS and child seat to my order - it seemed like something was working incorrectly. I even accidentally submitted the reservation, as I was trying to find that total.Basically, the website was not user-friendly. Items were difficult to find, and the interface was confusing."It was hard to understand what the total cost would be with the add-ons included.Finding the ExtrasI saw no area to add the extrasI saw no way to add GPS or child seat.Because I didn’t know at what point it would offer the add-ons, and the total didn’t include the gps and car seat.It wasn’t clear where to add the extras and once they were added it wasn’t clear what the final total would be. I also thought payment details needed to be added before an actual booking would take place and they weren’t.I couldn’t find the address for either task. I couldn’t figure out the options until I noticed them after I chose the car.. Measuring Usability LLC
    22. 22. Reasons for Low Task Ratings on Renting a Car Error Bars are 90% Confidence Intervals Measuring Usability LLC
    23. 23. Total Price Problem C0xflQ%3d%3d Measuring Usability LLC
    24. 24. Finding Location & Search Results Measuring Usability LLC
    25. 25. Usability Problems: Renting a Car Extras Total Search ProblemUser Measuring Usability LLC
    26. 26. User by Problem Matrices Rent a Car Find a Location ProblemUser P = .12 P = .15 P = .09 P = .07 Measuring Usability LLC
    27. 27. Long Tail Problems file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d Measuring Usability LLC
    28. 28. Default Location Measuring Usability LLC
    29. 29. 1st Click Analysis Measuring Usability LLC
    30. 30. First Click Analysis • If a user’s first click is down the wrong-path, only 46% eventually succeed • If a user’s first click is down the right-path, 87% eventually succeedWolfson, et al (2008) Contextual card sorting (or FirstClick testing): A new methodology forvalidating information architectures, UPA Proceedings Measuring Usability LLC
    31. 31. Find the Nearest Location: 83% Complete42% Of Users84% Success50% Of Users80% Success 3% Of Users 100% Success Measuring Usability LLC
    32. 32. Find the Nearest Location: 71% Complete8% Of Users100% Success40% Of Users 10% Of Users71% Success 83% Success10% Of Users50% Success Measuring Usability LLC
    33. 33. Test Level Metrics Measuring Usability LLC
    34. 34. Attitude before and After Usability Test VeryFavorableUn-FavorableVery Diff % After users completed the tasks Budget’s Score increased 12% & Enterprises decreased 15% * P < .05 ; Error Bars are 90% Confidence Intervals and Paired t-test Used for Statistical Comparison Measuring Usability LLC
    35. 35. Net Promoter ScoreHow likely is it that you’ll recommend this website to a friend? Enterprise Budget 30th Percentile of Websites 73rd Percentile of Websites Measuring Usability LLC
    36. 36. Measuring Usability LLC
    37. 37. SUPR-Q and Sub-Scales Measuring Usability LLC
    38. 38. Takeaways & Tips• Pretest Tasks and Study Design• Within Subjects• Compared to What?• Triangulate with Multiple Metrics• Use Standardized Questions – SEQ – SUPR-Q• Remote Unmoderated Testing w/ Video – In Parallel – As a Subset Measuring Usability LLC
    39. 39. Find out More Sept 19-20th 2012 Hyatt Convention Center Denver, CO @LeanUXDenver Twitter @MsrUsability Measuring Usability LLC