Statistics for UX Professionals - Jessica Cameron

Statistics for UX Professionals
Jessica Cameron
UX Scotland
Thursday, 14 June 2017
1

Presentation title / Footer text 2Photo by Lane Jackman on Unsplash

3
Statistics for UX Professionals
Featuring more mean words
per slide than any other talk
you will see this week
@jessscameron

4
In the next 60 minutes, you will learn:
1. Useful statistical concepts, like what a confidence
interval is and how a margin of error is kind of the
same thing as a confidence interval, but not
completely
2. How many people you really need to fill out a survey
3. What kinds of quantitative data may be collected
during a usability test
4. How to report quantitative data collected during a
usability test
5. How not to report quantitative data collected during a
usability test

Presentation title / Footer text 5Photo by Roman Mager on Unsplash

6Photo by Carlos Muza on Unsplash
Useful statistical concepts

7
Why use statistics?
• Statistics let us estimate what everyone might do (a
population) by looking at what some people do (a
sample)
• Do not use statistics if you have access to the full
population (for example, all members of a project
team)

8
Key concepts
• Central tendency and spread
• Mean
• Standard deviation
• Outlier
• Confidence interval
• Confidence level
• Margin of error

9
Central tendency and spread
• Central tendency refers to an average, middle, or
typical value
• Spread refers to how spread out or squeezed
together a set of values are (relative to the central
tendency)
• Example (1 = not at all satisfied, 5 = very satisfied):
• Sample 1: 100 people rate a website 1, 100 people rate it 5
• Sample 2: 200 people rate a website 3
• How do the central tendency and spread differ?

10
Mean
• A mean is a measure of central tendency
• The mean is the sum of all measurements divided by
the number of measurements
• Also known as the average or arithmetic mean
• Can be computed in Excel using the AVERAGE
function
• A sample mean is an estimate of the true population
mean

11
How much do you like or not like spaghetti? (n=75)
0 1 2 3 4 5 6 7 8 9 10
I hate spaghetti I love spaghetti

12
Mean = 5.87
0 1 2 3 4 5 6 7 8 9 10

13
Standard deviation
• A measure of spread
• To find the standard deviation, find the square root of
the average of the squared differences between each
measurement and the mean
• Or use the STDEV function in Excel
• If the standard deviation is low, the measurements
are grouped close to the mean – and if it is high, they
are further apart
• If data are normally distributed, 95% of
measurements can be expected to fall within 2
standard deviations of the mean

15
How much do you like or not like spaghetti? (n=75)
0 1 2 3 4 5 6 7 8 9 10

16
Mean = 5.87
0 1 2 3 4 5 6 7 8 9 10

17
1 standard deviation = 3.29
0 1 2 3 4 5 6 7 8 9 10

18
2 standard deviations = 6.58
0 1 2 3 4 5 6 7 8 9 10

19
How much do you like or not like spaghetti?
0 1 2 3 4 5 6 7 8 9 10

20
Mean = 6.85
0 1 2 3 4 5 6 7 8 9 10

21
1 standard deviation = 2.58
0 1 2 3 4 5 6 7 8 9 10

22
2 standard deviations = 2.58
0 1 2 3 4 5 6 7 8 9 10

23
Outlier
• Any measurement that falls more than 2 standard
deviations from the mean
• May also be an unrealistic measurement (for
example, time spent on a task that is shorter than the
minimum time required by an expert user)
• Might represent valid, if extreme, data – such as an
important subgroup of users
• If 95% of measurements are within 2 standard
deviations of the mean, we would expect 5% of the
sample to be outliers

24
Any outliers here?
0 1 2 3 4 5 6 7 8 9 10

25
Confidence interval
• A measure of spread and sample size
• A confidence interval is a range of values that we can
say with a certain level of confidence contains the
true population mean
• This true population mean is the one we would find if
we were able to have all possible users perform and/
or rate a task
• Use the CONFIDENCE.T function in Excel
• (Use alpha = 0.05, which corresponds to a 95% level
of confidence)

26
95% CI (n=75): 6.26, 7.45
0 1 2 3 4 5 6 7 8 9 10

27
What determines the size of the
confidence interval?
• Standard deviation: The larger the standard deviation,
the more spread out data are and the larger the
confidence interval will be
• Sample size: The larger the sample size, the more
reliable measurements are and the smaller the

28
95% CI (n=75): 6.26, 7.45
0 1 2 3 4 5 6 7 8 9 10

29
95% CI (n=375): 6.59, 7.11
0 1 2 3 4 5 6 7 8 9 10
= 5 people

30
What determines the size of the
interval?
• Standard deviation: The larger the standard deviation,
the more spread out data are and the larger the
margin of error will be
• Sample size: The larger the sample size, the more
reliable measurements are and the smaller the
• Confidence level: The higher the level of confidence
you need, the larger the confidence interval will have
to be

31
Confidence level
• Confidence intervals are usually reported with a 95%
level of confidence
• That means that if you repeated a survey 100 times,
you would expect the mean response you get to fall
within your confidence interval 95 times
• You can be 95% confident that the true population
mean is within this interval as well
• 90% and 99% are also commonly accepted – and the
higher the level of confidence, the larger the resulting

32
95% CI (n=375): 6.59, 7.11
0 1 2 3 4 5 6 7 8 9 10

33
95% CI (n=375): 6.51, 7.20
0 1 2 3 4 5 6 7 8 9 10

34
Confidence interval – how to report
Wrong:
• We are 95% confident that the population mean is 6.85
Right:
• We can say with 95% confidence that the true population
mean lies between 6.51 and 7.20
What is the difference between a confidence interval and
a margin of error?

35
Margin of error
• A margin of error is half of a confidence interval
• Excel actually gives us the margin of error, which we
then use to compute the confidence interval
• A margin of error is usually half of a 95% confidence
interval, with the confidence level implied
• (That means that it is easier to talk about margin of
error than confidence interval)
• Margin of error is often used for poll numbers or
preferences, and is expressed in percentage points

36
Margin of error notes
• The ‘error’ refers to random sampling error
• It does not account for other errors:
• Systematic sampling errors
• Errors in survey design (e.g., biased or unclear questions)
• Each measurement has a margin of error associated
with it
• Poll shows Trump at 17% and a bag of hammers at 81%, with
a +/- 3 point margin of error
• Trump could really get anywhere between 14% and 20% of
the vote, and the bag of hammers could get anywhere
between 78% and 84% of the vote

37
Recap of key concepts
• Central tendency and spread
• Mean
• Standard deviation
• Outlier
• Confidence interval
• Confidence level
• Margin of error

38
Statistics for surveys
Photo by Ranier Ridao on Unsplash

39
How many survey participants?
(Short answer)
400

40
How many survey participants?
(Long answer)
• It depends on how confident you want to be in the
findings (e.g., what size confidence interval are you
comfortable with?)
• It depends on who your audience is
• It depends on the size of the population you are
drawing your sample from

41
How many participants?
(Long answer)
• Examples of population sizes:
• Everyone in the UK = 60 million
• Topshop Oxford Street shoppers = 200,000 per week
• Topshop.com shoppers = 4.5 million per week
• Topshop employees = 10,000
• A ‘large’ population is anything over about 20,000
• Most surveys of existing (or potential) customers will
draw from a large population

42
Confidence level = 95%
+/- 5
Margin of error
+/- 3 +/- 1
Population size
100 80 92 99
500 217 341 475
1,000 278 517 906
10,000 370 965 4,899
20,000 377 1,014 6,489
50,000 382 1,045 8,057
100,000 383 1,056 8,762
500,000 384 1,065 9,423
1,000,000 384 1,066 9,512

43
95% CI (n=75): 6.26, 7.45
0 1 2 3 4 5 6 7 8 9 10

44
95% CI (n=375): 6.59, 7.11
0 1 2 3 4 5 6 7 8 9 10

45
Collecting vs. comparing data
• Surveys typically collect satisfaction data for an
existing website at a given time
• For that, mean + confidence interval (or margin of
error) is an effective way of communicating results
• But what if you are comparing two sets of data?
• Satisfaction ratings before and after a launch
• Conversion rates on versions A and B of a site
• Then you will need a proper statistical test

46
Use a t-test
• To compare two samples, use a two sample t-test
• A t-test will tell you if the populations that those
samples come from are the same or different (to a
certain level of confidence)
• It does this by considering the difference between the
means relative to the size of the variance, or spread
• Not enough to see if confidence levels overlap
• That would be an overly conservative test
• So instead of eyeballing the data, run a t-test

47
How to run a t-test
• Use the T.TEST function in Excel – need to know
arrays, tails and type
• The two arrays are the two data sets
• The number of tails depends on what you are predicting
• For type, you can generally assume equal variances (type =
2)
• Result is a probability that the samples come from the
same population
• If it is less than .05, you can say that the populations
are different

48
Mean = 6.85
0 1 2 3 4 5 6 7 8 9 10

49
Mean = 5.87
0 1 2 3 4 5 6 7 8 9 10

50
0 1 2 3 4 5 6 7 8 9 10
T-test (N=150): p = .043

51Photo by Sandeep Swarnkar on Unsplash

52
Rules of thumb
• At a 95% confidence level:
• A sample size of 400 will yield a margin of error of +/- 5%
• A sample size of 1,000 will yield a margin of error of +/- 3%
• If you are serious enough about your research to use
the recommended number of participants, then you
should take your data seriously
• Report central tendency and spread
• Look for statistically significant results

Statistics for user testing
53Photo by Rob Hampson on Unsplash

54
Data collected during usability tests
Qualitative data:
• Observations of ease or difficulty participants have
completing tasks
• What we think about when we think about usability
testing
Quantitative data:
• Success/ failure/ disaster rates (effectiveness)
• Task completion time (efficiency)
• User surveys (satisfaction)

55
Satisfaction data collected during
usability tests
• Do you collect satisfaction ratings (or any other scale
ratings) from usability test participants?
• How many people do you usually run during a single
usability test?
• How do you report those data?
• How should you report those data?

56
Asking about satisfaction
• Questions asked during usability testing can tell us a
great deal about the experience that our participants
are having on a website
• They can tell us basically nothing about the
experience that all users have on a website

57
Remember confidence intervals?
• Your sample size is 6
• Three testers (50%) say they would use the website
again
• You can say with 95% confidence that the true
percentage of the population who would use this
website again is somewhere between…
• 10% and 90%

58
So be careful
• Be careful not only with what you choose to ask…
• But how you choose to report the data
• Graphs and charts can be very compelling…
• But so can qualitative data
https://finickypenguin.wordpress.com/category/nerd-
humor/page/3/

59
Takeaways
• Reporting confidence intervals along with means
gives everyone more confidence in your findings
• Try to get 400 people to fill out your surveys
• You can use Excel to run simple statistical tests, but if
you need something more complicated try R or Stata
• Statistics are not the answer to everything…
• But if you choose to use them, they can expand your
UX toolkit nicely

Presentation title / Footer text 60Photo by Morvanic Lee on Unsplash
Jessica Cameron
@jessscameron
User Vision
@UserVision

Statistics for UX Professionals - Jessica Cameron

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Statistics for UX Professionals - Jessica Cameron

Similar to Statistics for UX Professionals - Jessica Cameron (20)

More from User Vision

More from User Vision (20)

Recently uploaded

Recently uploaded (20)

Statistics for UX Professionals - Jessica Cameron