Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Best Practices for Remote Usability
Testing
Comparative Benchmark Results

Jeff Sauro | Measuring Usability LLC
2/28/2012
5 Challenges Facing UX Professionals Today




1.   Time
2.   Cost
3.   Tools & Techniques
4.   Representative Users
5.   ...
Measuring Usability LLC
Deliverables




Business Requirement Doc Personas               Wireframes
Function Requirement Spec Usability Test Plan ...
Representative




                 Measuring Usability LLC
Unmoderated Remote Testing w/ Video: Not a
Panacea but pretty close




                                  Measuring Usabil...
Remote Testing Setup using UserZoom & Usertesting

                                                                    Web...
Case Study:




              Measuring Usability LLC
Summary
•   61 Total Remote Unmoderated Users
•   4 Tasks
•   Within Subjects
•   Randomized Tasks & Websites
•   Recruitm...
2 Tasks
1. Find the Nearest Location
Find the address of the nearest rental office to the Hilton Hotel located at 921 SW
 ...
Task Completion Rates




Error Bars are 90% Confidence Intervals
* p <. 01 ; N = 62
                                     ...
Reasons for Failure by Task




                              Measuring Usability LLC
Disaster Rates (Highest Confidence but Failure)




                                     Measuring Usability LLC
Task-Level Ease : Single Ease Question (SEQ)
Very Easy
Very Hard




            Error Bars are 90% Confidence Intervals
 ...
Task Time: Completed Tasks Only
Seconds




          # Users

          Error Bars are 90% Confidence Intervals
         ...
Single Usability Metric (SUM)




Avg




      Error Bars are 90% Confidence Intervals
      * p < . 01
                 ...
Completed Times: Find Nearest Location




   Median Time with all Users: 119
   Median Time Removed Long Times: 118


   ...
Why did the user take 7 minutes?




s.userzoom.com/m/MSBDMjUxUzM_ - Survey taker
                                    Meas...
Why?




       Measuring Usability LLC
Unmoderated Testing with Video




                                 Measuring Usability LLC
Verbatims
Totaling
It didn’t give the whole total with the extras-I had to add that myself.
Took the price of the car and ...
Reasons for Low Task Ratings on Renting a Car




   Error Bars are 90% Confidence Intervals

                            ...
Total Price Problem




http://accounts.usertesting.com/ViewClip.aspx?file=GSiU1W9elfLZjKxh
   C0xflQ%3d%3d

             ...
Finding Location & Search Results




                                    Measuring Usability LLC
Usability Problems: Renting a Car
 Extras Total Search        Problem
User




                                           ...
User by Problem Matrices
        Rent a Car           Find a Location
         Problem
User




          P = .12         ...
Long Tail Problems




http://accounts.usertesting.com/ViewClip.aspx?
  file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d
              ...
Default Location




                   Measuring Usability LLC
1st Click Analysis




                     Measuring Usability LLC
First Click Analysis

 • If a user’s first click is down the wrong-path,
   only 46% eventually succeed

 • If a user’s fi...
Find the Nearest Location: 83% Complete


42% Of Users
84% Success

50% Of Users
80% Success
                             ...
Find the Nearest Location: 71% Complete




8% Of Users
100% Success

40% Of Users
                                       ...
Test Level Metrics




                     Measuring Usability LLC
Attitude before and After Usability Test
     Very
Favorable
Un-Favorable
Very




          Diff %

        After users c...
Net Promoter Score
How likely is it that you’ll recommend this website to a friend?




            Enterprise            ...
Usability
Credibility
Loyalty
Appearance

www.SUPRQ.com

                Measuring Usability LLC
SUPR-Q and Sub-Scales




                        Measuring Usability LLC
Takeaways & Tips
•   Pretest Tasks and Study Design
•   Within Subjects
•   Compared to What?
•   Triangulate with Multipl...
Find out More




                                   www.LeanUXDenver.com
                                     Sept 19-20t...
Upcoming SlideShare
Loading in …5
×

of

Best practices for remote usability testing Slide 1 Best practices for remote usability testing Slide 2 Best practices for remote usability testing Slide 3 Best practices for remote usability testing Slide 4 Best practices for remote usability testing Slide 5 Best practices for remote usability testing Slide 6 Best practices for remote usability testing Slide 7 Best practices for remote usability testing Slide 8 Best practices for remote usability testing Slide 9 Best practices for remote usability testing Slide 10 Best practices for remote usability testing Slide 11 Best practices for remote usability testing Slide 12 Best practices for remote usability testing Slide 13 Best practices for remote usability testing Slide 14 Best practices for remote usability testing Slide 15 Best practices for remote usability testing Slide 16 Best practices for remote usability testing Slide 17 Best practices for remote usability testing Slide 18 Best practices for remote usability testing Slide 19 Best practices for remote usability testing Slide 20 Best practices for remote usability testing Slide 21 Best practices for remote usability testing Slide 22 Best practices for remote usability testing Slide 23 Best practices for remote usability testing Slide 24 Best practices for remote usability testing Slide 25 Best practices for remote usability testing Slide 26 Best practices for remote usability testing Slide 27 Best practices for remote usability testing Slide 28 Best practices for remote usability testing Slide 29 Best practices for remote usability testing Slide 30 Best practices for remote usability testing Slide 31 Best practices for remote usability testing Slide 32 Best practices for remote usability testing Slide 33 Best practices for remote usability testing Slide 34 Best practices for remote usability testing Slide 35 Best practices for remote usability testing Slide 36 Best practices for remote usability testing Slide 37 Best practices for remote usability testing Slide 38 Best practices for remote usability testing Slide 39
Upcoming SlideShare
Enterprise_CS
Next

6 Likes

Share

Best practices for remote usability testing

Slides from the webinar on Best Practices for Remote User Testing, led by a real guru on measuring UX, Jeff Sauro. Jeff presents best practices and a real case study that shows how to use UserZoom and UserTesting simultaneously for capturing quantitative and qualitative data.

Related Books

Free with a 30 day trial from Scribd

See all

Best practices for remote usability testing

  1. 1. Best Practices for Remote Usability Testing Comparative Benchmark Results Jeff Sauro | Measuring Usability LLC 2/28/2012
  2. 2. 5 Challenges Facing UX Professionals Today 1. Time 2. Cost 3. Tools & Techniques 4. Representative Users 5. Deliverables Measuring Usability LLC
  3. 3. Measuring Usability LLC
  4. 4. Deliverables Business Requirement Doc Personas Wireframes Function Requirement Spec Usability Test Plan User Screener Task Analysis Doc Recruitment Doc Usability Report Measuring Usability LLC
  5. 5. Representative Measuring Usability LLC
  6. 6. Unmoderated Remote Testing w/ Video: Not a Panacea but pretty close Measuring Usability LLC
  7. 7. Remote Testing Setup using UserZoom & Usertesting Website is shown in main window Users are shown the task and asked to complete it. Full A/V available online. Collects Task Time, Completion Rates, Task-Ease and Click Paths Measuring Usability LLC
  8. 8. Case Study: Measuring Usability LLC
  9. 9. Summary • 61 Total Remote Unmoderated Users • 4 Tasks • Within Subjects • Randomized Tasks & Websites • Recruitment Criteria: > 25 and US Based – 40 % Had Rented a Car from Budget or Enterprise • Mix of Age, Geography • 58% Female • Total Study Length : 20-30 minutes • Users Recruited and Data collected in 48 Hours Measuring Usability LLC
  10. 10. 2 Tasks 1. Find the Nearest Location Find the address of the nearest rental office to the Hilton Hotel located at 921 SW Sixth Avenue, Portland, Oregon, United States 97204. Write down or copy the name of the street of the nearest location. 2. Renting a Car Location: Logan Airport, Boston MA Rental Period : Fri April, 13th 2012 at 11.00am to Sunday April 15th at 5.00pm. Class: Intermediate (NO SUV’s) Extras: GPS Navigation and Car Seat for a 2 Year old If asked, use the following identification: Name : John Smith Email: john112233@mailinator.com Phone : 303-555-1212 Credit Card: Visa Measuring Usability LLC
  11. 11. Task Completion Rates Error Bars are 90% Confidence Intervals * p <. 01 ; N = 62 Measuring Usability LLC
  12. 12. Reasons for Failure by Task Measuring Usability LLC
  13. 13. Disaster Rates (Highest Confidence but Failure) Measuring Usability LLC
  14. 14. Task-Level Ease : Single Ease Question (SEQ) Very Easy Very Hard Error Bars are 90% Confidence Intervals * p < . 01 ; N = 62 Measuring Usability LLC
  15. 15. Task Time: Completed Tasks Only Seconds # Users Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
  16. 16. Single Usability Metric (SUM) Avg Error Bars are 90% Confidence Intervals * p < . 01 Measuring Usability LLC
  17. 17. Completed Times: Find Nearest Location Median Time with all Users: 119 Median Time Removed Long Times: 118 Measuring Usability LLC
  18. 18. Why did the user take 7 minutes? s.userzoom.com/m/MSBDMjUxUzM_ - Survey taker Measuring Usability LLC
  19. 19. Why? Measuring Usability LLC
  20. 20. Unmoderated Testing with Video Measuring Usability LLC
  21. 21. Verbatims Totaling It didn’t give the whole total with the extras-I had to add that myself. Took the price of the car and manually added on 3 day price of car seat ($10.99*3) and same for GPS but still unsure of total because unsure if these items are taxed. I could not get a total price of the rental reservation, and the price I got was not a possible answer on the survey. It told me that the price did not include the add-ons, and that those *may* be charged at the counter. I don’t know what it means by may be charged at the counter- does that mean they many not be charged for? I don’t understand why it doesn’t give a total cost of the rental with the GPS and toddler seat included. I added them in myself with a calculator, but I’m thought it would just automatically show me what the total will be. I was unable to find the link to add the GPS. I did read that it is 10.99 per day. I was also not able to find the link to add the car seat. you didn’t give an exact total for the booking. you said that the additional charges would be done at the counter. i think you should have gave me an estimated total so i wouldn’t have had to compute the charges myself. I just really wasn’t sure of my final pricing or even if I booked it right. I had to use a calculator to add the GPS and car seat charges to the car rental. I would have preferred if I could have selected the add-ons, and then it would show the total revised price before I booked the reservation. I was wondering if the add-ons would also have other charges or taxes. overall easier than the last one. Hard to know if updated price when adding the GPS and car seat "The website was confusing to me. I didn’t understand why the rental amount did not display an increase after I added the GPS and child seat to my order - it seemed like something was working incorrectly. I even accidentally submitted the reservation, as I was trying to find that total.Basically, the website was not user-friendly. Items were difficult to find, and the interface was confusing." It was hard to understand what the total cost would be with the add-ons included. Finding the Extras I saw no area to add the extras I saw no way to add GPS or child seat. Because I didn’t know at what point it would offer the add-ons, and the total didn’t include the gps and car seat. It wasn’t clear where to add the extras and once they were added it wasn’t clear what the final total would be. I also thought payment details needed to be added before an actual booking would take place and they weren’t. I couldn’t find the address for either task. I couldn’t figure out the options until I noticed them after I chose the car. . Measuring Usability LLC
  22. 22. Reasons for Low Task Ratings on Renting a Car Error Bars are 90% Confidence Intervals Measuring Usability LLC
  23. 23. Total Price Problem http://accounts.usertesting.com/ViewClip.aspx?file=GSiU1W9elfLZjKxh C0xflQ%3d%3d Measuring Usability LLC
  24. 24. Finding Location & Search Results Measuring Usability LLC
  25. 25. Usability Problems: Renting a Car Extras Total Search Problem User Measuring Usability LLC
  26. 26. User by Problem Matrices Rent a Car Find a Location Problem User P = .12 P = .15 P = .09 P = .07 Measuring Usability LLC
  27. 27. Long Tail Problems http://accounts.usertesting.com/ViewClip.aspx? file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d Measuring Usability LLC
  28. 28. Default Location Measuring Usability LLC
  29. 29. 1st Click Analysis Measuring Usability LLC
  30. 30. First Click Analysis • If a user’s first click is down the wrong-path, only 46% eventually succeed • If a user’s first click is down the right-path, 87% eventually succeed Wolfson, et al (2008) Contextual card sorting (or FirstClick testing): A new methodology for validating information architectures, UPA Proceedings Measuring Usability LLC
  31. 31. Find the Nearest Location: 83% Complete 42% Of Users 84% Success 50% Of Users 80% Success 3% Of Users 100% Success Measuring Usability LLC
  32. 32. Find the Nearest Location: 71% Complete 8% Of Users 100% Success 40% Of Users 10% Of Users 71% Success 83% Success 10% Of Users 50% Success Measuring Usability LLC
  33. 33. Test Level Metrics Measuring Usability LLC
  34. 34. Attitude before and After Usability Test Very Favorable Un-Favorable Very Diff % After users completed the tasks Budget’s Score increased 12% & Enterprises decreased 15% * P < .05 ; Error Bars are 90% Confidence Intervals and Paired t-test Used for Statistical Comparison Measuring Usability LLC
  35. 35. Net Promoter Score How likely is it that you’ll recommend this website to a friend? Enterprise Budget 30th Percentile of Websites 73rd Percentile of Websites Measuring Usability LLC
  36. 36. Usability Credibility Loyalty Appearance www.SUPRQ.com Measuring Usability LLC
  37. 37. SUPR-Q and Sub-Scales Measuring Usability LLC
  38. 38. Takeaways & Tips • Pretest Tasks and Study Design • Within Subjects • Compared to What? • Triangulate with Multiple Metrics • Use Standardized Questions – SEQ – SUPR-Q • Remote Unmoderated Testing w/ Video – In Parallel – As a Subset Measuring Usability LLC
  39. 39. Find out More www.LeanUXDenver.com Sept 19-20th 2012 Hyatt Convention Center Denver, CO @LeanUXDenver MeasuringUsability.com jeff@measuringusability.com Twitter @MsrUsability Measuring Usability LLC
  • JohnChurchwell

    Mar. 26, 2013
  • poebus

    Oct. 9, 2012
  • zachidin

    Sep. 30, 2012
  • alanho

    May. 21, 2012
  • CDUX

    Mar. 9, 2012
  • vafoodscaper

    Feb. 29, 2012

Slides from the webinar on Best Practices for Remote User Testing, led by a real guru on measuring UX, Jeff Sauro. Jeff presents best practices and a real case study that shows how to use UserZoom and UserTesting simultaneously for capturing quantitative and qualitative data.

Views

Total views

3,955

On Slideshare

0

From embeds

0

Number of embeds

1,189

Actions

Downloads

11

Shares

0

Comments

0

Likes

6

×