Combining Quantitative & Qualitative Data in a Single Large scale User Research Study

Measuring Usability LLC
Combining Quantitative &
Qualitative Data in a Single
Large-scale User Research
Study
Measuring Usability
Airline Benchmark Data

Jeff Sauro
Principal
Measuring Usability
Speaker
Alfonso de la Nuez
Co-Founder and Co-CEO
UserZoom
Moderator
Speakers:

Meet UserZoom!
All-in-One Software Solution to Cost-effectively
Measure the Digital Customer Experience and
Conduct Remote User Testing

• Increase conversion rates by conducting cost-
effective UX research, remote usability testing
and voice of the customer studies
• Test any web-based products (live or prototype)
and mobile apps, on a unified software platform.
Meet UserZoom!
• Founded in 2007; launched SaaS platform in 2009
• Our mission: To go beyond the Lab!
• International company: Silicon Valley, Spain, U.K., Germany
• 200 Enterprise customers, 50% of Fortune’s Most Admired Brands
• 90% renewal rate, 40% yoy revenue growth, profitable business.

• Participants take the study simultaneously
• ‘In-the-wild’ testing (natural context)
• Geographic representation
• No moderation needed
• Many types of studies can be conducted
• Output combines Usability + Behavior + Survey Feedback
What is Remote Unmoderated User Testing (URUT)
Participant
UZ Browser Bar

Select Clients using URUT

1) Cost-effective
• No moderation needed
• Automated data analysis
2) Agile & Efficient
• Build studies and gather
feedback in days
3) Qual + Quant
• Success rates, time on task,
behavior, video, audio,
heatmaps, verbatims,
dendrograms, etc.
4) Easy Recruiting
• Geographic representation
• No travel needed
5) Feature Rich
• More flexible, customizable,
robust, versitile than others
6) Team & Service
• Hands-on, highly experienced
Customer Success Team
Why is UZ unique?

Study Results

RESEARCH APPROACH
Tentative
Hypothesis
Observation
Induction Deduction
Pattern
Theory
Hypothesis

Qualitative Quantitative

STUDY STRUCTURE
Retrospective SurveyN = 1000
N = 130
Asked about their most
recent experience
Randomly assigned 2 websites,
asked to make a reservation
SessionVideo and Audio
Task Based Study

AIRLINE WEBSITES INCLUDED
N = 105
N = 121
N = 111
N = 110
N = 105
N = 102
N = 103
N = 107
DATES
N = 108

SUPR-Q & FACTORS
Standardized User Experience Percentile Rank Questionnaire Post Experience
USABILITY
• This website is easy to use.
• It is easy to navigate within the website.
LOYALTY
• How likely are you to recommend this website to a
friend or colleague?
• I will likely visit this website in the future.
APPEARANCE
• I find this website to have a clean and simple
presentation.
• I find the website to be attractive.
CREDIBILITY (trust, value & comfort)
• I feel comfortable purchasing from this website.
• I feel confident conducting business with this website.
NPS
PROVIDES STANDARDIZED SCORES
RELATIVETO 200 OTHER WEBSITES 15
See Appendix for More Details

OVERALL SCORE BY SITE
Industry Average = 83%
Error Bars are 90% confidence intervals
91%
83%
65%
82%
90%
84%
87%
88%
79%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Poor Excellent

APPEARANCE BY SITE
Industry Average = 75%Error Bars are 90% confidence intervals
82%
82%
35%
87%
78%
73%
83%
83%
77%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I find this website to have a clean and simple presentation.
I find the website to be attractive.
Poor Excellent

Qualitative Insights

UNITED APPEARANCE : HOMEPAGE
What one thing would you improve on the United.com website? 41% MentionedAppearance
UserComments
• “Make the home page simpler, not have
so much information condensed in such
a small space.”
• “Cleaner look.There are so many
different colors used on the main page
that it’s distracting to look at & takes
longer to figure out which part of the
site is relevant.”
• “It’s very busy on the home page.... lots
and lots of tabs and place to click”
• “The homepage looks extremely
cluttered. It is daunting.”

Task Based Insights

STUDY LOGISTICS
webcam microphone

VALIDITY OF FINDINGS
Internal Validity
External Validity

ENCOURAGE USERSTOTHINK ALOUD

TASK BASED SUMMARY
The success rates were essentially the same across all three sites
82% 84% 83%
CONFIDENCE
+
EASE
The experience across the three different websites were
different, as seen in the confidence and ease scores
Southwest was preferred 2:1 over United, further supporting the
findings from the quantitative study
American had the worst task performance of the three sites
tested – Flight selection and seat upgrades are overwhelming

LEVEL OF EASE
5.88 5.81 5.28
1
2
3
4
5
6
7
United Southwest American
Overall, how difficult or easy was it to complete this task [make a reservation] ?
99%
89%
46%
20%
9%
4%
2%
Average level of ease
Very Easy
Very
Difficult
American had significantly lower ease.

CONFIDENCE
6.00 6.05 5.43
1
2
3
4
5
6
7
United Southwest American
How confident are you that you completed the task successfully [making a reservation] ?
99%
43%
12%
4%
2%
1%
.3%
Average Confidence
Very
Confident
Not
Confident
At All

AMERICAN: FLIGHT SELECTION FRICTION
This user seems confused when attempting to select a return flight and continues to select upgrades to their
departing flight instead before ultimately giving up.

AMERICAN FLIGHT SELECTION

AMERICAN : DIFFICULTY MAKING RESERVATION
“Sometime actually too much info, I
wish it would just give me top 5 flights
within my time/price parameters with
the option to expand if it doesn’t meet
my requirements.”
“Although I like the layout, I am not a
huge fan of the box in the middle of
the page. It is the one feature of the
website that makes it a little hard to
read.”
“The website layout is
sort of confusing.”
Seat upgrade/upsell is confusing users

UNITED SEGMENT PRICING CONFUSION

UNITED: SEGMENT PRICING
Large Study User: “Search results for flights can be confusing; I like seeing departing flights from origin city and
destination city at the same time. It’s unclear what the final price will be because it gives individual price points for
each flight, yet these prices aren't actually combined for a final cost.”

UNITED: CHANGING RESERVATIONS
“when you are in the part where you are deciding on dates for a flight - to be able to change the dates without
starting the search over would be nice”
“It should be easier to modify search criteria without going back and forth.”
United had the lowest overall SUPRQ score as well as the lowest Usability Score. Southwest andAmerican
have the highest Usability scores, where you are able to change your travel dates from the search results.

UNITED: STARTING SEARCH OVER

INTERACTION SATISFACTION
Making Res. Calendar Speed Checking In Contact Changing Res. Choosing Seat Using Miles
Sample Mean – Overall Mean
Standard deviation
= Z Score = Percentile Rank
5.7 - 5.9
.2
-1.02= = 15%
Accuracy
Variability

INTERACTIONS: AIRLINES
94%
81%67% 45%48% 26% 22% 5%
97%97% 62%77% 25% 19% 47%
69% 53%72% 19%15% 50% 15% 6%
70% 32%52% 9%13% 2% 0% 3%
Green = Positive Interaction Red = Negative Interaction Orange = Neither strong negative or positive interaction

UNITED: CALENDAR
The calendar feature on the
United website was rated as the
hardest to use.
The Calendar does not
automatically pop up when you
click in the date box, therefore
making some users manually
enter the dates, which leads to
error

SOUTHWEST: CALENDAR
United’s calendar was rated as being significantly harder than Southwest’s calendar.
When a user tabs through the details for their flight, the calendar automatically pops up, eliminating the
thought that you would have to enter the dates manually, leading to a successful flight search.

0% 5% 10% 15% 20% 25%
Delta
Southwest
United
American
Orbitz
Travelocity
Kayak
Priceline
Expedia
Booked the Wrong Date
RECALLED MISTAKES
Average 10%
American Airlines users had significantly more mistakes than all other websites (p=.1).

LOYALTY & USABILITY
Website usability explains 35% of likelihood to recommend scores
Net Promoter, NPS, and Net Promoter Score are trademarks of Satmetrix Systems, Inc., Bain & Company, and Fred Reichheld
Usability % Rank: 18% 69% 99%
How likely are you to recommend to a colleague or friend?
0 1 2 3 4 5 6 7 8 9 10
Detractors Passives Promoters
Not at all Likely Neutral Extremely Likely

KEY DRIVERS OF UX QUALITY

KEY DRIVERS BY SITETYPE
Leverage & ExpandDisasters
Fix Maintain

KEY DRIVERS: AIRLINES
Choosing a Seat
94%
Changing Reservation
22%
+
-
Checking In
66%
Finding Contact Info
28%
Checking In
92%
Interacting with Calendar
7%
Changing Reservation
0%
Checking In
22%

SUMMARY & CONCLUSIONS
We’ve gone from general to particular and particular to general in
understanding the user experience.
Videos reinforced and illustrated quantitative patterns
Southwest had high scores and United had low
scores in both studies. Some evidence for lurking problems with
American Airlines.
Think aloud audio was essential in understanding some issues
Induction Deduction
Key drivers of UX differ by airline

@MeasuringU
UX Boot Camp
Denver CO, Aug 20th-22nd 2014
3 Days of Hands-On Training on User
Experience Methods, Metrics and Analysis.
Participants receive a User Experience
Research Certificate and 2.5 Continuing
Education Credits from the University of
Denver.
DenverUX.com
Measuring Usability LLC is a usability and quantitative research firm
based in Denver, Colorado USA focusing on the statistical analysis of
human behavior and quantifying the user experience for Fortune 1000
companies. www.MeasuringUsability.com
48
MEASURING USABILITY

Appendix

• The SUPR-Q is a set of 8 standardized questions that gauge the usability, credibility, user
loyalty, and appearance of a website
• SUPR scores are reported as a percentile ranking out of a database of over 200 websites
• For example, a score of 50% means that half of all website rank higher and half rank lower
• The 8 SUPR questions were selected after testing 48 candidate questions on thousands of
users and dozens of websites
• The SUPR-Q usability factor exhibits strong correlation with the System Usability Scale
(SUS)
• SUPR rankings are more useful than SUS scores because they offer a comparison with other
websites
More information is available online at www.suprq.com
UNDERSTANDINGTHE SUPR-Q

Combining Quantitative & Qualitative Data in a Single Large scale User Research Study

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to Combining Quantitative & Qualitative Data in a Single Large scale User Research Study

Similar to Combining Quantitative & Qualitative Data in a Single Large scale User Research Study (20)

More from UserZoom

More from UserZoom (20)

Recently uploaded

Recently uploaded (20)

Combining Quantitative & Qualitative Data in a Single Large scale User Research Study

Editor's Notes