Webinar slides: Scale your UX Research and Convince Stakeholders with UZ Session Replay

967 views

Published on

In this webinar, Jeff Sauro and Alfonso de la Nuez discuss how to:

- Combine both quant plus qual data and benchmark UX
- Be leaner and cost-effective in your research
- Tell a complete story about UX and convince your stakeholders with both stats and video recordings

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
967
On SlideShare
0
From Embeds
0
Number of Embeds
146
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • August 1st, 1981 – MTV broadcasted their first music video
  • Nps stayed the same for edmunds --- findability is still the issue (and price quotes) but NPS went from -5 to 48 for KBB
  • Kbb p=.17
  • How difficult is this task across a number of websites?Price illustrates edmunds price quoteToyota-Build illustrates that it is even hard on Toyota.comHow much of the difficulty is finding the right car?Autotrader and Cars.com especiallyCars:_find car illustrates thisAuto_pricehybri: illustrates thisCars-get price: illustrates thisHow much is it the interaction with the payment calculator?Car_price: illustrates not being able to select a car from calculatorKBB price ca: illustrates not being able to select a car from calcHow much is it the problem with the task design? Ed-task design: (on 2nd study) illustrates this
  • Nps
  • Webinar slides: Scale your UX Research and Convince Stakeholders with UZ Session Replay

    1. 1. UserZoom Session Replay An Automotive Case Study Jeff Sauro | Measuring Usability LLC & UserZoom 2/26/13 #uzwebinarMeasuring Usability LLC
    2. 2. Quick intro to UserZoom & UZ Session Replay • All-in-one Enterprise software solution that helps Businesses cost-effectively test, measure and improve UX over websites & mobile apps. • Specialists in online or remote UX research & testing • Scale your UX research, save time, money, effort • While still obtaining very rich insights • 45 employees, offices in Sunnyvale (CA) Manchester (UK), Munich (DE) and Barcelona (Spain) • Work for 50% of Fortune’s Most Admired BrandsMeasuring Usability LLC
    3. 3. About Remote Unmoderated User Testing• Like online surveys… ‘on steroids’• Test large samples, from all over the place• Natural context• Ask users to complete tasks, sort items, click here…• Track usability metrics (success rates, time on task), satisfaction, and also behavior• Lots of automation, lots of time saving, $ saving too• Lots of metrics, charts, user feedback, etc. Measuring Usability LLC
    4. 4. About Remote Unmoderated User TestingMeasuring Usability LLC
    5. 5. About Remote Unmoderated User Testing How about VIDEO?Measuring Usability LLC
    6. 6. Introducing UZ Session Replay• You can collect UX metrics such as task success ratios, time on task, number of clicks and number of unique pages viewed…• …and now, with UZ Session Replay, in the same study you can also get video recording of each user session. • UX Storytelling just got a whole lot better! • Combine both QUANT + QUAL • The Lab, at times, just doesn’t seem to add that much value • Filter videos by task, questionnaire answers or success/fail per task, bookmark important events or actions, create highlight clips for your reports, record particular URLs or block particular URLs from recording (https for example), Share the videos with othersMeasuring Usability LLC
    7. 7. Introducing UZ Session ReplayMeasuring Usability LLC
    8. 8. Introducing UZ Session ReplayMeasuring Usability LLC
    9. 9. Measuring Usability LLC 9
    10. 10. Measuring Usability LLC 10
    11. 11. 1981 Toyota CamryMeasuring Usability LLC 11
    12. 12. 2013 Toyota CamryMeasuring Usability LLC 12
    13. 13. Background and Purpose• Finding the monthly payment for a car is hard. – Task Completion rate < 40% – Harder than ~80% of all tasks• Why? What causes that? – Is it site specific? – Is this just a difficult task? • Multi-step of finding the right car with the right features • Then calculating the payments – Is it both or something else?Measuring Usability LLC 13
    14. 14. Find the Monthly Payment for a 2013 Toyota Camry $485 $477 $458 $492 $458 102 Users Each Randomly Assigned 3 Websites How much would your monthly payment be for a 2013 Toyota Camry Hybrid LE Series in zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a 8.75% sales tax.Measuring Usability LLC 14
    15. 15. Monthly Payment Completion Rates 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 35% 38% 17% 19% 33% 0% Edmunds KBB Cars.com Autotrader Toyota The 2 part task was made harder on Cars.com and Autotrader because finding the price of the basic model was more difficultMeasuring Usability LLC 15
    16. 16. Task Level Ease (SEQ 1-7 Standardized Question) Extremely Easy 100% 90% 80% 70% 60% 50% 40% 30% Extremely Difficult 20% 10% 22% 29% 31% 10% 22% 0% Edmunds KBB Car.com Autotrader ToyotaUsers rated Cars.com higher than the others in terms of ease, even though Cars.com had the lowest completion rates. Cars.com gave users a false sense of confidence Measuring Usability LLC 16
    17. 17. Disasters [High Confidence & Task Failure] 100% 9% 9% 13% 13% 90% 24% 80% 70% 52% 49% 57% 60% 72% Disaster 50% Error 59% Success 40% 30% 20% 38% 35% 33% 10% 19% 17% 0% Edmunds KBB Cars.com Autotrader Toyota Cars.com has the highest disaster rate, users felt a false sense of confidence on Cars.comMeasuring Usability LLC 17
    18. 18. Edmunds VideosMeasuring Usability LLC 18
    19. 19. Edmunds (phase 1) 33% Completion Rate 7% 11% 8% 8% 4% 40% Measuring Usability LLC 19
    20. 20. Why was this task hard? – Edmunds Categorized open ended responses “couldn’t find a way to calculate a monthly payment without entering contact information for a call-back or email”Measuring Usability LLC 20
    21. 21. To The Other VideosMeasuring Usability LLC 21
    22. 22. KBB (phase 1) 38% Completion Rate 10%17% 13% Measuring Usability LLC 22
    23. 23. Why was this task hard – KBB Categorized open ended responses 90% “KBB didn’t let me 80% choose a specific model/make on the payments estimator 70% page. I had to first search for the Toyota 60% Camry Hybrid LE” 50% 40% 30% 20% 10% 69% 19% 13% 0% Hard me finding car/price of car Non-Intui ve website Could not find calculatorMeasuring Usability LLC 23
    24. 24. Why was this task hard? – Cars.com Categorized open ended responses 80% “Website didn’t let me choose the car I 70% wanted to finance (or I couldn’t find that) and I had to guess at 60% the purchase price” 50% 40% 30% 20% 10% 50% 25% 15% 10% 0% Difficulty finding price of the Hard Task/Did not know where Slow Loading Time Could not find the calculator car to startMeasuring Usability LLC 24
    25. 25. Why was this task difficult? – Autotrader Categorized open ended responses 60% finding the hybrid model was difficult 50% 40% 30% 20% 10% 41% 32% 14% 8% 5% 0% Hard to find the price of Could not find the Hard website to Navigate A lot of steps Did not want to enter the car calculator personal informa onMeasuring Usability LLC 25
    26. 26. Why was this task difficult? – Toyota Categorized open ended responses 60% The Toyota website seemed to slow my computer down considerably. 50% 40% 30% 20% 10% 30% 26% 22% 13% 9% 0% Slow website/hard to Calculator was hard to Could not find price Could not find Could not find price of navigate use without dealers quote calculator carMeasuring Usability LLC 26
    27. 27. Phase 2 : Eliminate the Car Finding 57 Users Each Attempted the Same Task on Both Websites (Randomized) Phase 1 How much would your monthly payments be for a 2013 Toyota Camry Hybrid LE Series in zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a 8.75% sales tax. Phase 2 How much would your monthly payments be for a 2013 Toyota Camry Hybrid LE Series priced at $28,500 in zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a 8.75% sales tax.Measuring Usability LLC 27
    28. 28. Completion Rates (Phase 2) 80% 70% 60% 50% 40% Original Task Given Price 30% 20% 10% 35% 57% 38% 53% 0% Edmunds KBB The completion rate for Edmunds went up 22% points when the participant was given the price of the car. It went up 15% for KBBMeasuring Usability LLC 28
    29. 29. Task Level Ease (Phase 2) Extremely Easy 70% 60% 50% 40% Original Task 30% Given Price 20% Extremely Difficult 10% 22% 20% 29% 47% 0% Edmunds KBBKBB is perceived to be much easier when the price is given because they do not give you the option to select your vehicle from the calculator like Edmunds. Instead, if you navigate to the calculator first, you then have to search for your car price, therefore eliminating that step increases the task ease. Measuring Usability LLC 29
    30. 30. Why was this task Difficult? – Edmunds (Phase 2) Categorized open ended responses 90% “I wasn’t shown any tools to help me 80% calculate my monthly payment. I had to do it on my own.” 70% 60% 50% 40% 30% 20% 10% 68% 23% 9% 0% Hard to Find Calculator Site was Confusing Popups Finding the calculator and getting coaxed by new car quotes is what continues to make this a hard taskMeasuring Usability LLC 30
    31. 31. Time on Task Success Only (Phase 2) 360 300 240 180 Original Task Given Price 120 60 252 253 227 108 0 Edmunds * KBB In Phase 2, the time it took to complete the task on KBB dropped significantly and statisticallyMeasuring Usability LLC 31
    32. 32. Edmunds (phase 2) 57% Completion Rate 2% 11% 9% 8% 51% Measuring Usability LLC 32
    33. 33. KBB (phase 2) 52% Completion Rate 31%13% 4% Measuring Usability LLC 33
    34. 34. Summary Benefits of having Video 1. Having a Record of the Interaction 2. Verify Task Completion By relaxing the success criteria the completion rate went from 47% - 57% 3. Understand Causes of Failure Where and why? How much of the difficulty is finding the right car? Between 15 and 22 percentage points: Still only ~50% completing How much is it the problem with the task design? Slight changes to the calculator swing monthly payments Changing Zip Codes and a specific car users aren’t interested in are artificialMeasuring Usability LLC 34
    35. 35. About Jeff Sauro & Measuring Usability LLC Measuring Usability LLC is a usability and quantitative research firm based in Denver, Colorado USA focusing on the statistical analysis of human behavior and quantifying the user experience for Fortune 1000 companies. DenverUX.com Sept 18-20th www.MeasuringUsability.com @MeasuringUMeasuring Usability LLC 35
    36. 36. AppendixMeasuring Usability LLC 36
    37. 37. Website’s Capabilities The website’s capabilities meet my requirements. Kbb 3.44 Cars.com 3.33 Edmunds 3.23 Toyota 3.07 Autotrader 2.56 1 2 3 4 5 Completely Disagree Completely AgreeMeasuring Usability LLC 37
    38. 38. Time on Task – Success Only 420 360 300 240 180 120 60 252 227 219 144 156.5 0 Edmunds KBB Cars Autotrader Toyota Autotrader was the fastest, though not statisticallyMeasuring Usability LLC 38
    39. 39. Experimental Study• How difficult is this task across a number of websites? – Across all the websites we looked at, there were low completion rates, proving that across platforms, it is a difficult task • Edmunds everyone gets distracted by the Price Quote pop-ups and pages • KBB, Cars.com and Autotrader do not let you select the car from the calculator • Toyota has a lot of graphics, but is hard to navigate through• How much of the difficulty is finding the right car? – The two step process of finding the right car and then finding the calculator adds to the difficulty of the task. On the websites that are more geared towards car inventory instead of research, finding the basic price of the car to input into the calculator is difficult• How much is it the interaction with the payment calculator? – It is less the interaction with inputting figures into the calculator that is difficult, instead it is when you cannot choose a car FROM the calculator that makes this step of the task difficult. – Due to the artificiality of the task, uses have ore trouble entering in the correct figures than if they were doing this on their own. A small typo or mistake would look like a task failure• How much is it the problem with the task design? – More just the artificiality of if you were going to search for a car, you would not be changing things like your zip code or your sales tax• Is there a mismatch with the users mental model?Measuring Usability LLC 39
    40. 40. SUPR-Q Items & Factors Usability This website is easy to use. It is easy to navigate within the website. Credibility (Trust, Value & Comfort) This website keeps the promises it makes to me. I feel confident conducting business with this website. Loyalty How likely are you to recommend this website to a friend or colleague? NPS I will likely visit this website in the future. Appearance I found the website to be attractive. The website has a clean and simple presentation.Measuring Usability LLC 40
    41. 41. SUPRQ & NPS (Phase 1) 60% 40% 20% 19% 6% 10% 1% 10% 0% Edmunds KBB Cars.com Autotrader Toyota -5% -20% -38% SUPRQ -41% -52% NPS -63% -40% -60% -80% -100% Across all sites, users have low NPS and low SUPR-Q scores which illustrates that this is a difficult task on any website.Measuring Usability LLC 41

    ×