• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Webinar slides: Scale your UX Research and Convince Stakeholders with UZ Session Replay
 

Webinar slides: Scale your UX Research and Convince Stakeholders with UZ Session Replay

on

  • 673 views

In this webinar, Jeff Sauro and Alfonso de la Nuez discuss how to: ...

In this webinar, Jeff Sauro and Alfonso de la Nuez discuss how to:

- Combine both quant plus qual data and benchmark UX
- Be leaner and cost-effective in your research
- Tell a complete story about UX and convince your stakeholders with both stats and video recordings

Statistics

Views

Total Views
673
Views on SlideShare
538
Embed Views
135

Actions

Likes
2
Downloads
0
Comments
0

1 Embed 135

http://info.userzoom.com 135

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • August 1st, 1981 – MTV broadcasted their first music video
  • Nps stayed the same for edmunds --- findability is still the issue (and price quotes) but NPS went from -5 to 48 for KBB
  • Kbb p=.17
  • How difficult is this task across a number of websites?Price illustrates edmunds price quoteToyota-Build illustrates that it is even hard on Toyota.comHow much of the difficulty is finding the right car?Autotrader and Cars.com especiallyCars:_find car illustrates thisAuto_pricehybri: illustrates thisCars-get price: illustrates thisHow much is it the interaction with the payment calculator?Car_price: illustrates not being able to select a car from calculatorKBB price ca: illustrates not being able to select a car from calcHow much is it the problem with the task design? Ed-task design: (on 2nd study) illustrates this
  • Nps

Webinar slides: Scale your UX Research and Convince Stakeholders with UZ Session Replay Webinar slides: Scale your UX Research and Convince Stakeholders with UZ Session Replay Presentation Transcript

  • UserZoom Session Replay An Automotive Case Study Jeff Sauro | Measuring Usability LLC & UserZoom 2/26/13 #uzwebinarMeasuring Usability LLC
  • Quick intro to UserZoom & UZ Session Replay • All-in-one Enterprise software solution that helps Businesses cost-effectively test, measure and improve UX over websites & mobile apps. • Specialists in online or remote UX research & testing • Scale your UX research, save time, money, effort • While still obtaining very rich insights • 45 employees, offices in Sunnyvale (CA) Manchester (UK), Munich (DE) and Barcelona (Spain) • Work for 50% of Fortune’s Most Admired BrandsMeasuring Usability LLC
  • About Remote Unmoderated User Testing• Like online surveys… ‘on steroids’• Test large samples, from all over the place• Natural context• Ask users to complete tasks, sort items, click here…• Track usability metrics (success rates, time on task), satisfaction, and also behavior• Lots of automation, lots of time saving, $ saving too• Lots of metrics, charts, user feedback, etc. Measuring Usability LLC
  • About Remote Unmoderated User TestingMeasuring Usability LLC
  • About Remote Unmoderated User Testing How about VIDEO?Measuring Usability LLC
  • Introducing UZ Session Replay• You can collect UX metrics such as task success ratios, time on task, number of clicks and number of unique pages viewed…• …and now, with UZ Session Replay, in the same study you can also get video recording of each user session. • UX Storytelling just got a whole lot better! • Combine both QUANT + QUAL • The Lab, at times, just doesn’t seem to add that much value • Filter videos by task, questionnaire answers or success/fail per task, bookmark important events or actions, create highlight clips for your reports, record particular URLs or block particular URLs from recording (https for example), Share the videos with othersMeasuring Usability LLC
  • Introducing UZ Session ReplayMeasuring Usability LLC
  • Introducing UZ Session ReplayMeasuring Usability LLC
  • Measuring Usability LLC 9
  • Measuring Usability LLC 10
  • 1981 Toyota CamryMeasuring Usability LLC 11
  • 2013 Toyota CamryMeasuring Usability LLC 12
  • Background and Purpose• Finding the monthly payment for a car is hard. – Task Completion rate < 40% – Harder than ~80% of all tasks• Why? What causes that? – Is it site specific? – Is this just a difficult task? • Multi-step of finding the right car with the right features • Then calculating the payments – Is it both or something else?Measuring Usability LLC 13
  • Find the Monthly Payment for a 2013 Toyota Camry $485 $477 $458 $492 $458 102 Users Each Randomly Assigned 3 Websites How much would your monthly payment be for a 2013 Toyota Camry Hybrid LE Series in zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a 8.75% sales tax.Measuring Usability LLC 14
  • Monthly Payment Completion Rates 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 35% 38% 17% 19% 33% 0% Edmunds KBB Cars.com Autotrader Toyota The 2 part task was made harder on Cars.com and Autotrader because finding the price of the basic model was more difficultMeasuring Usability LLC 15
  • Task Level Ease (SEQ 1-7 Standardized Question) Extremely Easy 100% 90% 80% 70% 60% 50% 40% 30% Extremely Difficult 20% 10% 22% 29% 31% 10% 22% 0% Edmunds KBB Car.com Autotrader ToyotaUsers rated Cars.com higher than the others in terms of ease, even though Cars.com had the lowest completion rates. Cars.com gave users a false sense of confidence Measuring Usability LLC 16
  • Disasters [High Confidence & Task Failure] 100% 9% 9% 13% 13% 90% 24% 80% 70% 52% 49% 57% 60% 72% Disaster 50% Error 59% Success 40% 30% 20% 38% 35% 33% 10% 19% 17% 0% Edmunds KBB Cars.com Autotrader Toyota Cars.com has the highest disaster rate, users felt a false sense of confidence on Cars.comMeasuring Usability LLC 17
  • Edmunds VideosMeasuring Usability LLC 18
  • Edmunds (phase 1) 33% Completion Rate 7% 11% 8% 8% 4% 40% Measuring Usability LLC 19
  • Why was this task hard? – Edmunds Categorized open ended responses “couldn’t find a way to calculate a monthly payment without entering contact information for a call-back or email”Measuring Usability LLC 20
  • To The Other VideosMeasuring Usability LLC 21
  • KBB (phase 1) 38% Completion Rate 10%17% 13% Measuring Usability LLC 22
  • Why was this task hard – KBB Categorized open ended responses 90% “KBB didn’t let me 80% choose a specific model/make on the payments estimator 70% page. I had to first search for the Toyota 60% Camry Hybrid LE” 50% 40% 30% 20% 10% 69% 19% 13% 0% Hard me finding car/price of car Non-Intui ve website Could not find calculatorMeasuring Usability LLC 23
  • Why was this task hard? – Cars.com Categorized open ended responses 80% “Website didn’t let me choose the car I 70% wanted to finance (or I couldn’t find that) and I had to guess at 60% the purchase price” 50% 40% 30% 20% 10% 50% 25% 15% 10% 0% Difficulty finding price of the Hard Task/Did not know where Slow Loading Time Could not find the calculator car to startMeasuring Usability LLC 24
  • Why was this task difficult? – Autotrader Categorized open ended responses 60% finding the hybrid model was difficult 50% 40% 30% 20% 10% 41% 32% 14% 8% 5% 0% Hard to find the price of Could not find the Hard website to Navigate A lot of steps Did not want to enter the car calculator personal informa onMeasuring Usability LLC 25
  • Why was this task difficult? – Toyota Categorized open ended responses 60% The Toyota website seemed to slow my computer down considerably. 50% 40% 30% 20% 10% 30% 26% 22% 13% 9% 0% Slow website/hard to Calculator was hard to Could not find price Could not find Could not find price of navigate use without dealers quote calculator carMeasuring Usability LLC 26
  • Phase 2 : Eliminate the Car Finding 57 Users Each Attempted the Same Task on Both Websites (Randomized) Phase 1 How much would your monthly payments be for a 2013 Toyota Camry Hybrid LE Series in zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a 8.75% sales tax. Phase 2 How much would your monthly payments be for a 2013 Toyota Camry Hybrid LE Series priced at $28,500 in zip code 90210 with a $2000 down payment, a 3.89% interest rate over 5 years with a 8.75% sales tax.Measuring Usability LLC 27
  • Completion Rates (Phase 2) 80% 70% 60% 50% 40% Original Task Given Price 30% 20% 10% 35% 57% 38% 53% 0% Edmunds KBB The completion rate for Edmunds went up 22% points when the participant was given the price of the car. It went up 15% for KBBMeasuring Usability LLC 28
  • Task Level Ease (Phase 2) Extremely Easy 70% 60% 50% 40% Original Task 30% Given Price 20% Extremely Difficult 10% 22% 20% 29% 47% 0% Edmunds KBBKBB is perceived to be much easier when the price is given because they do not give you the option to select your vehicle from the calculator like Edmunds. Instead, if you navigate to the calculator first, you then have to search for your car price, therefore eliminating that step increases the task ease. Measuring Usability LLC 29
  • Why was this task Difficult? – Edmunds (Phase 2) Categorized open ended responses 90% “I wasn’t shown any tools to help me 80% calculate my monthly payment. I had to do it on my own.” 70% 60% 50% 40% 30% 20% 10% 68% 23% 9% 0% Hard to Find Calculator Site was Confusing Popups Finding the calculator and getting coaxed by new car quotes is what continues to make this a hard taskMeasuring Usability LLC 30
  • Time on Task Success Only (Phase 2) 360 300 240 180 Original Task Given Price 120 60 252 253 227 108 0 Edmunds * KBB In Phase 2, the time it took to complete the task on KBB dropped significantly and statisticallyMeasuring Usability LLC 31
  • Edmunds (phase 2) 57% Completion Rate 2% 11% 9% 8% 51% Measuring Usability LLC 32
  • KBB (phase 2) 52% Completion Rate 31%13% 4% Measuring Usability LLC 33
  • Summary Benefits of having Video 1. Having a Record of the Interaction 2. Verify Task Completion By relaxing the success criteria the completion rate went from 47% - 57% 3. Understand Causes of Failure Where and why? How much of the difficulty is finding the right car? Between 15 and 22 percentage points: Still only ~50% completing How much is it the problem with the task design? Slight changes to the calculator swing monthly payments Changing Zip Codes and a specific car users aren’t interested in are artificialMeasuring Usability LLC 34
  • About Jeff Sauro & Measuring Usability LLC Measuring Usability LLC is a usability and quantitative research firm based in Denver, Colorado USA focusing on the statistical analysis of human behavior and quantifying the user experience for Fortune 1000 companies. DenverUX.com Sept 18-20th www.MeasuringUsability.com @MeasuringUMeasuring Usability LLC 35
  • AppendixMeasuring Usability LLC 36
  • Website’s Capabilities The website’s capabilities meet my requirements. Kbb 3.44 Cars.com 3.33 Edmunds 3.23 Toyota 3.07 Autotrader 2.56 1 2 3 4 5 Completely Disagree Completely AgreeMeasuring Usability LLC 37
  • Time on Task – Success Only 420 360 300 240 180 120 60 252 227 219 144 156.5 0 Edmunds KBB Cars Autotrader Toyota Autotrader was the fastest, though not statisticallyMeasuring Usability LLC 38
  • Experimental Study• How difficult is this task across a number of websites? – Across all the websites we looked at, there were low completion rates, proving that across platforms, it is a difficult task • Edmunds everyone gets distracted by the Price Quote pop-ups and pages • KBB, Cars.com and Autotrader do not let you select the car from the calculator • Toyota has a lot of graphics, but is hard to navigate through• How much of the difficulty is finding the right car? – The two step process of finding the right car and then finding the calculator adds to the difficulty of the task. On the websites that are more geared towards car inventory instead of research, finding the basic price of the car to input into the calculator is difficult• How much is it the interaction with the payment calculator? – It is less the interaction with inputting figures into the calculator that is difficult, instead it is when you cannot choose a car FROM the calculator that makes this step of the task difficult. – Due to the artificiality of the task, uses have ore trouble entering in the correct figures than if they were doing this on their own. A small typo or mistake would look like a task failure• How much is it the problem with the task design? – More just the artificiality of if you were going to search for a car, you would not be changing things like your zip code or your sales tax• Is there a mismatch with the users mental model?Measuring Usability LLC 39
  • SUPR-Q Items & Factors Usability This website is easy to use. It is easy to navigate within the website. Credibility (Trust, Value & Comfort) This website keeps the promises it makes to me. I feel confident conducting business with this website. Loyalty How likely are you to recommend this website to a friend or colleague? NPS I will likely visit this website in the future. Appearance I found the website to be attractive. The website has a clean and simple presentation.Measuring Usability LLC 40
  • SUPRQ & NPS (Phase 1) 60% 40% 20% 19% 6% 10% 1% 10% 0% Edmunds KBB Cars.com Autotrader Toyota -5% -20% -38% SUPRQ -41% -52% NPS -63% -40% -60% -80% -100% Across all sites, users have low NPS and low SUPR-Q scores which illustrates that this is a difficult task on any website.Measuring Usability LLC 41