This document summarizes two studies on usability testing. The first study examines how many users are needed to find most usability problems. It analyzes data from a test of 100 users and finds that testing 5 users can find 55-85% of problems, 10 users find 82-94% of problems, and 15 users find 90-97% of problems. The second study compares problems found through usability testing with 30 users versus expert reviews with 14 experts across 6 websites. It finds that only 14% of problems were identified by both users and experts, with users finding more navigation and content problems and experts finding more consistency and standards problems.

1. UXPA13 unconference
10 minutes.
2 studies.
Discuss.
Kath Straub PhD

2. Truth in advertising
I’m not talking about
my own work.
I’m talking about studies in the (peer-reviewed) literature
that you should know about and cite….

3. The question
How many users? Really.

4. The study people point to: Nielsen & Landauer (1993)
JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems.
ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213,
May1993,Amsterdam,TheNetherlands
4
# problems found: N(1-(1-λ)i)
N= Total # of usability problems
λ = probablity of finding the average usability problem when
running a single, average subject or using a single, average
evaluator
i= # of participants or evaluators
# of test participants or expert evaluators

5. The study people point to: Nielsen & Landauer (1993)
JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems.
ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213,
May1993,Amsterdam,TheNetherlands
5
Important
insight
If
you
don’t
test
any
users
or
have
anyone
review
the
usability,
you
won’t
learn
about
any
problems.
# of test participants or expert evaluators

6. The study people point to: Nielsen & Landauer (1993)
JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems.
ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213,
May1993,Amsterdam,TheNetherlands
6
You
hit
diminishing
returns
on
finding
new
problems
at
~
5par>cipants
or
5
evaluators
(assuming
each
user/evaluator
unearths
1/3
of
the
usability
problems)
# of test participants or expert evaluators

7. The real data (its in the paper ….) is messier.
JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems.
ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213,
May1993,Amsterdam,TheNetherlands
7
# of test participants or expert evaluators
This
shows
what
happens
when
you
play
with
the
assump>on
that
each
par>cipant/reviewer
finds
~
1/3
of
the
problems

8. Why this study should give us pause….
JakobNielsen,ThomasK.Landauer,Amathematicalmodelofthefindingofusabilityproblems.
ProceedingsoftheINTERCHI'93ConferenceonHumanFactorsinComputingSystems,p.206-213,
May1993,Amsterdam,TheNetherlands
8
• Its not really answering this particular question.
It is answering the question “Is there a
mathematical model that describes/predicts the
point of diminishing returns with test participants
or evaluators based on the post hoc analysis of 11
specific studies?
• The data is a lot messier than is typically cited
NB: This study is still worth reading, ….

9. 9
The study that answers the question:
How many users do I need to test to be confident
I will find most of the UX problems?
Faulkner, L. Beyond the five-user assumption:
Benefits of increased sample sizes in usability
testing. Behavior Research Methods, Instruments
& Computers, 35, 3, Psychonomic Society
(2003), 379--383.

10. What did Faulkner do?
10
Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting.
BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),
379--383.
1. Conducted a large scale (100 participant) usability test of
a corporate intranet (A real study, btw.)
2. Analyzed findings by test and by participant
EX: 100 participants uncovered 800 problems
Participant 1 uncovered 8 problems
Participant 2 uncovered 20 problems
Participant 3 uncovered 25 problems (NOTE: These are not the real #s)
3. Simulated usability tests by drawing repeated random sets
of participants from the pool of 100. Set sizes were 5, 10, 15,
20 …. 50 per test.
4. Plotted the (range of variability) of # problems found for
each test by participant groups size

11. Variability of findings in tests w/ 5 (random) users
11
%age
of
problems
found
Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting.
BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),
379--383.
Quite
a
few
outliers,
based
on
random
par4cipant
draws

12. Variability of findings in tests w/ 10 (random) users
12
Each
dot
is
a
simulated
“test”
%age
of
problems
found
Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting.
BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),
379--383.
A
few
outliers
here
…

13. Variability of findings in tests w/ 15 (random) users
13
%age
of
problems
found
Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting.
BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),
379--383.
Each
dot
is
a
simulated
“test”
No
outliers
here
…

14. The Δ in variability of problems found (Min – Max)
14
Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting.
BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),
379--383.
Practitioner Takeaway
If you test ….
• 5, you may observe between 55-85% of the problems, depending on the
luck of the participant draw.
• 10 you will capture between 82% - 94% of the problems
• 15 you get between 90-97% of the problems
How comfortable are you (or is your client) about the risk in testing small Ns?

15. 15
Faulkner,L.Beyondthefive-userassumption:Benefitsofincreasedsamplesizesinusabilitytesting.
BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),
379--383.
Practitioner Takeaway
If you test ….
• Only 5, statistically, you observe between 55-85% of the problems. How
many/few you observe depends on the luck of your participant draw.
• With 10 you will capture between 82% - 94% of the problems
• 15 you get between 90-97% of the problems
How comfortable are you (or is your client) about the risk of missing problems
because you are testing small Ns?

16. One typical response to Faulkner’s finding
But, but…, I’m a UX type
I do a heuristic review along the way.
Doesn’t that mitigate the risk?

17. Question #2
Do users care about what
UXers rant about?

18. The Nielsen heuristics ….
18
Do you have a scorecard?
Is it (even loosely)
based on this?

19. The Nielsen heuristics ….
19
Forget dog years.
How many years abo is
this in internet time?
Note that these are
(still!) useful. But at a
different level of
description.

20. 20
The study that answers the question:
Are the things that trip users up the same as the
things that UX types notice in expert review?
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA.

21. What did Petrie, et. al. do?
21
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
Compared
Usability Testing Findings (What users focus on)
6 websites - 30 users
Expert Review Findings (What UXers focus on)
14 experts* / 3 different ER strategies
• Collabora>ve
heuris>c
evalua>on
• Group
Usability
Expert
Walkthrough
• Group
Domain
Expert
Walkthrough
(DEW)
935 problems found

22. 4 categories of problems
22
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
Only 14%
of the identified
problems overlap??
Fourbroadcategoriesofproblems
χ2 is not significantly different for this table. That means that the
distribution of the number problems found in each category by only
users, only UXers or both is not different from what would be
predicted.

23. 23
% of problems in
that sub-category
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
Problems
that
UXers
and
users
both
no>ce
…

24. 24
Users uncovered 1.8 times as
many problems, so we would
expect this to be 1.8 : 1
Users : Experts
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
Problems
Users
only
reported
more
than
UXers
only

25. 25
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
Experts uncovered 1.8 times
fewer problems, so we expect
this ratio to be is 0.55 : 1
Experts : Users
Problems
UXers
only
reported
more
than
Users
only

26. 26
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
Problems
that
we
agree
about
…
Practitioner Takeaways
• We need to recognize that users care less and are less
hindered by some of the things we traditionaly obsess
about
• We need to be cautious about guidelines. We need
them, but we also need to recognize that things change
and guidelines need to be updated.
• Users can learn! As more users have become more experienced
with the web, they’ve learned some of the basic patterns … AND
some of the basic mistakes designers make. And they work around
them.
We must update our guidelines periodically, against real
use behaviors, in studies such as this one.

27. What does “Expert” mean for Petrie, et.al.?
27
Petrie,
H.
&
Power,
C.
(2012)
What
Do
Users
Really
Care
About?
A
Comparison
of
Usability
Problems
Found
by
Users
and
Experts
on
Highly
Interac>ve
Websites
CHI’12,
Aus>n,
Texas,
USA
14 usability experts
• 5 women / 9 men
• higher education qualifications or courses in HCI.
• 5+ years experience in usability
• Worked as professionals in user experience,
interaction or software/product design
• Usability responsibilities ranged from 50-100%
time
• 11 self-described as “experienced”; 3 as “junior”.
• Nearly all had conducted heuristic evaluations

28. QUESTIONS LATER ….?
28
kstraub
kath@usability.org
@kathstraub

29. Usability.org services include
• Customer Research
• User Experience Design/Digital strategy for Web, GUI and mobile
• Content in Plain Language
• Design for sustainable Behavioral Change
• Usability Evaluation (Data-driven continuous improvement)
We present hands-on training and mentoring in
• User research methods
• Interaction design for Web, GUI and Mobile
• Usability evaluation methods
• Content strategies and writing in plain language
• Workshop: Design strategies that drive behavioral change/Gamification
• Workshop: Research in Practice: Studies Usability Professionals should to
know about
* Many thanks to the Norwegian Tax Authority for allowing us to use their RiP workshop photo