Best Practices for Remote UsabilityTestingComparative Benchmark ResultsJeff Sauro | Measuring Usability LLC2/28/2012
5 Challenges Facing UX Professionals Today1.   Time2.   Cost3.   Tools & Techniques4.   Representative Users5.   Deliverab...
Measuring Usability LLC
DeliverablesBusiness Requirement Doc Personas               WireframesFunction Requirement Spec Usability Test Plan   User...
Representative                 Measuring Usability LLC
Unmoderated Remote Testing w/ Video: Not aPanacea but pretty close                                  Measuring Usability LLC
Remote Testing Setup using UserZoom & Usertesting                                                                    Websi...
Case Study:              Measuring Usability LLC
Summary•   61 Total Remote Unmoderated Users•   4 Tasks•   Within Subjects•   Randomized Tasks & Websites•   Recruitment C...
2 Tasks1. Find the Nearest LocationFind the address of the nearest rental office to the Hilton Hotel located at 921 SW    ...
Task Completion RatesError Bars are 90% Confidence Intervals* p <. 01 ; N = 62                                          Me...
Reasons for Failure by Task                              Measuring Usability LLC
Disaster Rates (Highest Confidence but Failure)                                     Measuring Usability LLC
Task-Level Ease : Single Ease Question (SEQ)Very EasyVery Hard            Error Bars are 90% Confidence Intervals         ...
Task Time: Completed Tasks OnlySeconds          # Users          Error Bars are 90% Confidence Intervals          * p < . ...
Single Usability Metric (SUM)Avg      Error Bars are 90% Confidence Intervals      * p < . 01                             ...
Completed Times: Find Nearest Location   Median Time with all Users: 119   Median Time Removed Long Times: 118            ...
Why did the user take 7 minutes?s.userzoom.com/m/MSBDMjUxUzM_ - Survey taker                                    Measuring ...
Why?       Measuring Usability LLC
Unmoderated Testing with Video                                 Measuring Usability LLC
VerbatimsTotalingIt didn’t give the whole total with the extras-I had to add that myself.Took the price of the car and man...
Reasons for Low Task Ratings on Renting a Car   Error Bars are 90% Confidence Intervals                                   ...
Total Price Problemhttp://accounts.usertesting.com/ViewClip.aspx?file=GSiU1W9elfLZjKxh   C0xflQ%3d%3d                     ...
Finding Location & Search Results                                    Measuring Usability LLC
Usability Problems: Renting a Car Extras Total Search        ProblemUser                                           Measuri...
User by Problem Matrices        Rent a Car           Find a Location         ProblemUser          P = .12                P...
Long Tail Problemshttp://accounts.usertesting.com/ViewClip.aspx?  file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d                     ...
Default Location                   Measuring Usability LLC
1st Click Analysis                     Measuring Usability LLC
First Click Analysis • If a user’s first click is down the wrong-path,   only 46% eventually succeed • If a user’s first c...
Find the Nearest Location: 83% Complete42% Of Users84% Success50% Of Users80% Success                                     ...
Find the Nearest Location: 71% Complete8% Of Users100% Success40% Of Users                                                ...
Test Level Metrics                     Measuring Usability LLC
Attitude before and After Usability Test     VeryFavorableUn-FavorableVery          Diff %        After users completed th...
Net Promoter ScoreHow likely is it that you’ll recommend this website to a friend?            Enterprise                  ...
UsabilityCredibilityLoyaltyAppearancewww.SUPRQ.com                Measuring Usability LLC
SUPR-Q and Sub-Scales                        Measuring Usability LLC
Takeaways & Tips•   Pretest Tasks and Study Design•   Within Subjects•   Compared to What?•   Triangulate with Multiple Me...
Find out More                                   www.LeanUXDenver.com                                     Sept 19-20th 2012...
Upcoming SlideShare
Loading in...5
×

Best practices for remote usability testing

2,701

Published on

Slides from the webinar on Best Practices for Remote User Testing, led by a real guru on measuring UX, Jeff Sauro. Jeff presents best practices and a real case study that shows how to use UserZoom and UserTesting simultaneously for capturing quantitative and qualitative data.

Published in: Technology, Business
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,701
On Slideshare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
11
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Parallel testingSolo UserZoomSolo Usertesting
  • SUM is the average of task completion rate, task-level satisfaction and the standardized task-time (from completed users only).You can interpret these numbers by saying “Renting a car on Budget has a usability score of 58% out of 100%”
  • Budget_Good_Enterprise Bad
  • Need 1 Users to have an 85% chance of detecting p = .80Need 56 Users for p = .033
  • Budget_Portland
  • Transcript of "Best practices for remote usability testing"

    1. 1. Best Practices for Remote UsabilityTestingComparative Benchmark ResultsJeff Sauro | Measuring Usability LLC2/28/2012
    2. 2. 5 Challenges Facing UX Professionals Today1. Time2. Cost3. Tools & Techniques4. Representative Users5. Deliverables Measuring Usability LLC
    3. 3. Measuring Usability LLC
    4. 4. DeliverablesBusiness Requirement Doc Personas WireframesFunction Requirement Spec Usability Test Plan User ScreenerTask Analysis Doc Recruitment Doc Usability Report Measuring Usability LLC
    5. 5. Representative Measuring Usability LLC
    6. 6. Unmoderated Remote Testing w/ Video: Not aPanacea but pretty close Measuring Usability LLC
    7. 7. Remote Testing Setup using UserZoom & Usertesting Website is shown in main window Users are shown the task and asked to complete it. Full A/V available online.Collects Task Time, Completion Rates, Task-Ease and Click Paths Measuring Usability LLC
    8. 8. Case Study: Measuring Usability LLC
    9. 9. Summary• 61 Total Remote Unmoderated Users• 4 Tasks• Within Subjects• Randomized Tasks & Websites• Recruitment Criteria: > 25 and US Based – 40 % Had Rented a Car from Budget or Enterprise• Mix of Age, Geography• 58% Female• Total Study Length : 20-30 minutes• Users Recruited and Data collected in 48 Hours Measuring Usability LLC
    10. 10. 2 Tasks1. Find the Nearest LocationFind the address of the nearest rental office to the Hilton Hotel located at 921 SW Sixth Avenue, Portland, Oregon, United States 97204. Write down or copy the name of the street of the nearest location.2. Renting a CarLocation: Logan Airport, Boston MARental Period : Fri April, 13th 2012 at 11.00am to Sunday April 15th at 5.00pm.Class: Intermediate (NO SUV’s)Extras: GPS Navigation and Car Seat for a 2 Year oldIf asked, use the following identification: Name : John Smith Email: john112233@mailinator.com Phone : 303-555-1212 Credit Card: Visa Measuring Usability LLC
    11. 11. Task Completion RatesError Bars are 90% Confidence Intervals* p <. 01 ; N = 62 Measuring Usability LLC
    12. 12. Reasons for Failure by Task Measuring Usability LLC
    13. 13. Disaster Rates (Highest Confidence but Failure) Measuring Usability LLC
    14. 14. Task-Level Ease : Single Ease Question (SEQ)Very EasyVery Hard Error Bars are 90% Confidence Intervals * p < . 01 ; N = 62 Measuring Usability LLC
    15. 15. Task Time: Completed Tasks OnlySeconds # Users Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
    16. 16. Single Usability Metric (SUM)Avg Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
    17. 17. Completed Times: Find Nearest Location Median Time with all Users: 119 Median Time Removed Long Times: 118 Measuring Usability LLC
    18. 18. Why did the user take 7 minutes?s.userzoom.com/m/MSBDMjUxUzM_ - Survey taker Measuring Usability LLC
    19. 19. Why? Measuring Usability LLC
    20. 20. Unmoderated Testing with Video Measuring Usability LLC
    21. 21. VerbatimsTotalingIt didn’t give the whole total with the extras-I had to add that myself.Took the price of the car and manually added on 3 day price of car seat ($10.99*3) and same for GPS but still unsure of total because unsure if these items are taxed.I could not get a total price of the rental reservation, and the price I got was not a possible answer on the survey.It told me that the price did not include the add-ons, and that those *may* be charged at the counter. I don’t know what it means by may be charged at the counter- does that mean they many not be charged for? I don’t understand why it doesn’t give a total cost of the rental with the GPS and toddler seat included. I added them in myself with a calculator, but I’m thought it would just automatically show me what the total will be.I was unable to find the link to add the GPS. I did read that it is 10.99 per day. I was also not able to find the link to add the car seat.you didn’t give an exact total for the booking. you said that the additional charges would be done at the counter. i think you should have gave me an estimated total so i wouldn’t have had to compute the charges myself.I just really wasn’t sure of my final pricing or even if I booked it right.I had to use a calculator to add the GPS and car seat charges to the car rental. I would have preferred if I could have selected the add-ons, and then it would show the total revised price before I booked the reservation. I was wondering if the add-ons would also have other charges or taxes. overall easier than the last one.Hard to know if updated price when adding the GPS and car seat"The website was confusing to me. I didn’t understand why the rental amount did not display an increase after I added the GPS and child seat to my order - it seemed like something was working incorrectly. I even accidentally submitted the reservation, as I was trying to find that total.Basically, the website was not user-friendly. Items were difficult to find, and the interface was confusing."It was hard to understand what the total cost would be with the add-ons included.Finding the ExtrasI saw no area to add the extrasI saw no way to add GPS or child seat.Because I didn’t know at what point it would offer the add-ons, and the total didn’t include the gps and car seat.It wasn’t clear where to add the extras and once they were added it wasn’t clear what the final total would be. I also thought payment details needed to be added before an actual booking would take place and they weren’t.I couldn’t find the address for either task. I couldn’t figure out the options until I noticed them after I chose the car.. Measuring Usability LLC
    22. 22. Reasons for Low Task Ratings on Renting a Car Error Bars are 90% Confidence Intervals Measuring Usability LLC
    23. 23. Total Price Problemhttp://accounts.usertesting.com/ViewClip.aspx?file=GSiU1W9elfLZjKxh C0xflQ%3d%3d Measuring Usability LLC
    24. 24. Finding Location & Search Results Measuring Usability LLC
    25. 25. Usability Problems: Renting a Car Extras Total Search ProblemUser Measuring Usability LLC
    26. 26. User by Problem Matrices Rent a Car Find a Location ProblemUser P = .12 P = .15 P = .09 P = .07 Measuring Usability LLC
    27. 27. Long Tail Problemshttp://accounts.usertesting.com/ViewClip.aspx? file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d Measuring Usability LLC
    28. 28. Default Location Measuring Usability LLC
    29. 29. 1st Click Analysis Measuring Usability LLC
    30. 30. First Click Analysis • If a user’s first click is down the wrong-path, only 46% eventually succeed • If a user’s first click is down the right-path, 87% eventually succeedWolfson, et al (2008) Contextual card sorting (or FirstClick testing): A new methodology forvalidating information architectures, UPA Proceedings Measuring Usability LLC
    31. 31. Find the Nearest Location: 83% Complete42% Of Users84% Success50% Of Users80% Success 3% Of Users 100% Success Measuring Usability LLC
    32. 32. Find the Nearest Location: 71% Complete8% Of Users100% Success40% Of Users 10% Of Users71% Success 83% Success10% Of Users50% Success Measuring Usability LLC
    33. 33. Test Level Metrics Measuring Usability LLC
    34. 34. Attitude before and After Usability Test VeryFavorableUn-FavorableVery Diff % After users completed the tasks Budget’s Score increased 12% & Enterprises decreased 15% * P < .05 ; Error Bars are 90% Confidence Intervals and Paired t-test Used for Statistical Comparison Measuring Usability LLC
    35. 35. Net Promoter ScoreHow likely is it that you’ll recommend this website to a friend? Enterprise Budget 30th Percentile of Websites 73rd Percentile of Websites Measuring Usability LLC
    36. 36. UsabilityCredibilityLoyaltyAppearancewww.SUPRQ.com Measuring Usability LLC
    37. 37. SUPR-Q and Sub-Scales Measuring Usability LLC
    38. 38. Takeaways & Tips• Pretest Tasks and Study Design• Within Subjects• Compared to What?• Triangulate with Multiple Metrics• Use Standardized Questions – SEQ – SUPR-Q• Remote Unmoderated Testing w/ Video – In Parallel – As a Subset Measuring Usability LLC
    39. 39. Find out More www.LeanUXDenver.com Sept 19-20th 2012 Hyatt Convention Center Denver, CO @LeanUXDenver MeasuringUsability.com jeff@measuringusability.com Twitter @MsrUsability Measuring Usability LLC

    ×