Your SlideShare is downloading. ×
0
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Combining Quantitative & Qualitative Data in a Single Large scale User Research Study
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Combining Quantitative & Qualitative Data in a Single Large scale User Research Study

667

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
667
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
24
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Video of user talking before davids video
  • This person is after david’s video. She is looking at two different departing flights, thinking that they are the departing flight and the returning flight and is confused/amazed how expensive it is.
  • Transcript

    1. Measuring Usability LLC Combining Quantitative & Qualitative Data in a Single Large-scale User Research Study Measuring Usability Airline Benchmark Data
    2. Jeff Sauro Principal Measuring Usability Speaker Alfonso de la Nuez Co-Founder and Co-CEO UserZoom Moderator Speakers:
    3. Measuring Usability LLC Meet UserZoom! All-in-One Software Solution to Cost-effectively Measure the Digital Customer Experience and Conduct Remote User Testing
    4. Measuring Usability LLC • Increase conversion rates by conducting cost- effective UX research, remote usability testing and voice of the customer studies • Test any web-based products (live or prototype) and mobile apps, on a unified software platform. Meet UserZoom! • Founded in 2007; launched SaaS platform in 2009 • Our mission: To go beyond the Lab! • International company: Silicon Valley, Spain, U.K., Germany • 200 Enterprise customers, 50% of Fortune’s Most Admired Brands • 90% renewal rate, 40% yoy revenue growth, profitable business.
    5. Measuring Usability LLC • Participants take the study simultaneously • ‘In-the-wild’ testing (natural context) • Geographic representation • No moderation needed • Many types of studies can be conducted • Output combines Usability + Behavior + Survey Feedback What is Remote Unmoderated User Testing (URUT) Participant UZ Browser Bar
    6. Measuring Usability LLC Select Clients using URUT
    7. Measuring Usability LLC 1) Cost-effective • No moderation needed • Automated data analysis 2) Agile & Efficient • Build studies and gather feedback in days 3) Qual + Quant • Success rates, time on task, behavior, video, audio, heatmaps, verbatims, dendrograms, etc. 4) Easy Recruiting • Geographic representation • No travel needed 5) Feature Rich • More flexible, customizable, robust, versitile than others 6) Team & Service • Hands-on, highly experienced Customer Success Team Why is UZ unique?
    8. Measuring Usability LLC Study Results
    9. Measuring Usability LLC
    10. Measuring Usability LLC RESEARCH APPROACH Tentative Hypothesis Observation Induction Deduction Pattern Theory Hypothesis
    11. Measuring Usability LLC Qualitative Quantitative
    12. Measuring Usability LLC STUDY STRUCTURE Retrospective SurveyN = 1000 N = 130 Asked about their most recent experience Randomly assigned 2 websites, asked to make a reservation SessionVideo and Audio Task Based Study
    13. Measuring Usability LLC RESEARCH APPROACH Tentative Hypothesis Observation Induction Deduction Pattern Theory Hypothesis
    14. Measuring Usability LLC AIRLINE WEBSITES INCLUDED N = 105 N = 121 N = 111 N = 110 N = 105 N = 102 N = 103 N = 107 DATES N = 108
    15. Measuring Usability LLC SUPR-Q & FACTORS Standardized User Experience Percentile Rank Questionnaire Post Experience USABILITY • This website is easy to use. • It is easy to navigate within the website. LOYALTY • How likely are you to recommend this website to a friend or colleague? • I will likely visit this website in the future. APPEARANCE • I find this website to have a clean and simple presentation. • I find the website to be attractive. CREDIBILITY (trust, value & comfort) • I feel comfortable purchasing from this website. • I feel confident conducting business with this website. NPS PROVIDES STANDARDIZED SCORES RELATIVETO 200 OTHER WEBSITES 15 See Appendix for More Details
    16. Measuring Usability LLC OVERALL SCORE BY SITE Industry Average = 83% Error Bars are 90% confidence intervals 91% 83% 65% 82% 90% 84% 87% 88% 79% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Poor Excellent
    17. Measuring Usability LLC APPEARANCE BY SITE Industry Average = 75%Error Bars are 90% confidence intervals 82% 82% 35% 87% 78% 73% 83% 83% 77% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% I find this website to have a clean and simple presentation. I find the website to be attractive. Poor Excellent
    18. Measuring Usability LLC Qualitative Insights
    19. Measuring Usability LLC
    20. Measuring Usability LLC
    21. Measuring Usability LLC
    22. Measuring Usability LLC UNITED APPEARANCE : HOMEPAGE What one thing would you improve on the United.com website? 41% MentionedAppearance UserComments • “Make the home page simpler, not have so much information condensed in such a small space.” • “Cleaner look.There are so many different colors used on the main page that it’s distracting to look at & takes longer to figure out which part of the site is relevant.” • “It’s very busy on the home page.... lots and lots of tabs and place to click” • “The homepage looks extremely cluttered. It is daunting.”
    23. Measuring Usability LLC Task Based Insights
    24. Measuring Usability LLC STUDY LOGISTICS webcam microphone
    25. Measuring Usability LLC
    26. Measuring Usability LLC VALIDITY OF FINDINGS Internal Validity External Validity
    27. Measuring Usability LLC ENCOURAGE USERSTOTHINK ALOUD
    28. Measuring Usability LLC TASK BASED SUMMARY The success rates were essentially the same across all three sites 82% 84% 83% CONFIDENCE + EASE The experience across the three different websites were different, as seen in the confidence and ease scores Southwest was preferred 2:1 over United, further supporting the findings from the quantitative study American had the worst task performance of the three sites tested – Flight selection and seat upgrades are overwhelming
    29. Measuring Usability LLC LEVEL OF EASE 5.88 5.81 5.28 1 2 3 4 5 6 7 United Southwest American Overall, how difficult or easy was it to complete this task [make a reservation] ? 99% 89% 46% 20% 9% 4% 2% Average level of ease Very Easy Very Difficult American had significantly lower ease.
    30. Measuring Usability LLC CONFIDENCE 6.00 6.05 5.43 1 2 3 4 5 6 7 United Southwest American How confident are you that you completed the task successfully [making a reservation] ? 99% 43% 12% 4% 2% 1% .3% Average Confidence Very Confident Not Confident At All
    31. Measuring Usability LLC AMERICAN: FLIGHT SELECTION FRICTION This user seems confused when attempting to select a return flight and continues to select upgrades to their departing flight instead before ultimately giving up.
    32. Measuring Usability LLC AMERICAN FLIGHT SELECTION
    33. Measuring Usability LLC AMERICAN : DIFFICULTY MAKING RESERVATION “Sometime actually too much info, I wish it would just give me top 5 flights within my time/price parameters with the option to expand if it doesn’t meet my requirements.” “Although I like the layout, I am not a huge fan of the box in the middle of the page. It is the one feature of the website that makes it a little hard to read.” “The website layout is sort of confusing.” Seat upgrade/upsell is confusing users
    34. Measuring Usability LLC UNITED SEGMENT PRICING CONFUSION
    35. Measuring Usability LLC UNITED: SEGMENT PRICING Large Study User: “Search results for flights can be confusing; I like seeing departing flights from origin city and destination city at the same time. It’s unclear what the final price will be because it gives individual price points for each flight, yet these prices aren't actually combined for a final cost.”
    36. Measuring Usability LLC UNITED: CHANGING RESERVATIONS “when you are in the part where you are deciding on dates for a flight - to be able to change the dates without starting the search over would be nice” “It should be easier to modify search criteria without going back and forth.” United had the lowest overall SUPRQ score as well as the lowest Usability Score. Southwest andAmerican have the highest Usability scores, where you are able to change your travel dates from the search results.
    37. Measuring Usability LLC UNITED: STARTING SEARCH OVER
    38. Measuring Usability LLC INTERACTION SATISFACTION Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles Sample Mean – Overall Mean Standard deviation = Z Score = Percentile Rank 5.7 - 5.9 .2 -1.02= = 15% Accuracy Variability
    39. Measuring Usability LLC INTERACTIONS: AIRLINES 94% Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles 81%67% 45%48% 26% 22% 5% 97%97% 62%77% 25% 19% 47% 69% 53%72% 19%15% 50% 15% 6% 70% 32%52% 9%13% 2% 0% 3% Green = Positive Interaction Red = Negative Interaction Orange = Neither strong negative or positive interaction
    40. Measuring Usability LLC UNITED: CALENDAR The calendar feature on the United website was rated as the hardest to use. The Calendar does not automatically pop up when you click in the date box, therefore making some users manually enter the dates, which leads to error
    41. Measuring Usability LLC SOUTHWEST: CALENDAR United’s calendar was rated as being significantly harder than Southwest’s calendar. When a user tabs through the details for their flight, the calendar automatically pops up, eliminating the thought that you would have to enter the dates manually, leading to a successful flight search.
    42. Measuring Usability LLC 0% 5% 10% 15% 20% 25% Delta Southwest United American Orbitz Travelocity Kayak Priceline Expedia Booked the Wrong Date RECALLED MISTAKES Average 10% American Airlines users had significantly more mistakes than all other websites (p=.1).
    43. Measuring Usability LLC LOYALTY & USABILITY Website usability explains 35% of likelihood to recommend scores Net Promoter, NPS, and Net Promoter Score are trademarks of Satmetrix Systems, Inc., Bain & Company, and Fred Reichheld Usability % Rank: 18% 69% 99% How likely are you to recommend to a colleague or friend? 0 1 2 3 4 5 6 7 8 9 10 Detractors Passives Promoters Not at all Likely Neutral Extremely Likely
    44. Measuring Usability LLC KEY DRIVERS OF UX QUALITY Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles
    45. Measuring Usability LLC KEY DRIVERS BY SITETYPE Leverage & ExpandDisasters Fix Maintain
    46. Measuring Usability LLC KEY DRIVERS: AIRLINES Choosing a Seat 94% Changing Reservation 22% + - Checking In 66% Finding Contact Info 28% Checking In 92% Interacting with Calendar 7% Changing Reservation 0% Checking In 22%
    47. Measuring Usability LLC SUMMARY & CONCLUSIONS We’ve gone from general to particular and particular to general in understanding the user experience. Videos reinforced and illustrated quantitative patterns Southwest had high scores and United had low scores in both studies. Some evidence for lurking problems with American Airlines. Think aloud audio was essential in understanding some issues Induction Deduction Key drivers of UX differ by airline
    48. Measuring Usability LLC @MeasuringU UX Boot Camp Denver CO, Aug 20th-22nd 2014 3 Days of Hands-On Training on User Experience Methods, Metrics and Analysis. Participants receive a User Experience Research Certificate and 2.5 Continuing Education Credits from the University of Denver. DenverUX.com Measuring Usability LLC is a usability and quantitative research firm based in Denver, Colorado USA focusing on the statistical analysis of human behavior and quantifying the user experience for Fortune 1000 companies. www.MeasuringUsability.com 48 MEASURING USABILITY
    49. Measuring Usability LLC Appendix
    50. Measuring Usability LLC • The SUPR-Q is a set of 8 standardized questions that gauge the usability, credibility, user loyalty, and appearance of a website • SUPR scores are reported as a percentile ranking out of a database of over 200 websites • For example, a score of 50% means that half of all website rank higher and half rank lower • The 8 SUPR questions were selected after testing 48 candidate questions on thousands of users and dozens of websites • The SUPR-Q usability factor exhibits strong correlation with the System Usability Scale (SUS) • SUPR rankings are more useful than SUS scores because they offer a comparison with other websites More information is available online at www.suprq.com UNDERSTANDINGTHE SUPR-Q

    ×