Loading...
Flash Player 9 (or above) is needed to view slideshows. We have detected that you do not have it on your computer.To install it, go here
Analysing quantitative data
An introduction to the analysis of quantitative data arising from user research
591 views | comments | 3 favorites | 80 downloads | 1 embeds (Stats)
More Info
This slideshow is Public
Total Views: 591 on Slideshare: 553 from embeds: 38
Most viewed embeds (Top 5):
More
Slideshow Transcript
- Slide 1: Analysing quantitative
data
with Steve Baty
UX Strategist
Web Directions User Experience ‘08
- Slide 2: Data is important
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 3: We expend
a lot of
effort to
gather it
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 4: We don’t always use it
well
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 5: We’ll be looking at:
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 6: We’ll be looking at:
* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 7: We’ll be looking at:
* time-to-completion
* task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 8: We’ll be looking at:
* time-to-completion
* task completion rates
* a/b testing
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 9: We’ll be looking at:
* time-to-completion
* task completion rates
* a/b testing
* page-view data
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 10: time-to-
completion
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 11: Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 12: 1 min 24 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 13: 1 min 23.8 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 14: 1 min 23.77 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 15: 1 min 23.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 16: 83.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 17: Our data Task 1
Task 2
User 1
83.5
131.1
User 2 User...
97.3
165.5
might Task 3
Task 4
Task 5
54.5
97.8
118.0
45.5
88.2
143.3
look like Task 6
Task 7
243.9
22.9
309.0
23.9
this...
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 18: We can calculate...
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 19: We can calculate...
mean - AVERAGE()
variance - VAR()
standard dev’n - STDEV()
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 20: Low-variability
Medium-variability
High-variability
90.43 s
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 21: Compare 2 sets of data
- between iterations
- between audience segments
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 22: Low sample sizes
restrict options
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 23: non-parametric
version == no
assumed dist’n
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 24: Rank-sum test
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 25: Time for a practical
demonstration
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 26: Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 27: Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 28: 1 3 3 3 5.5 5.5 7.5 7.5 9 10
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 29: 1 3 3 3 5.5 5.5 7.5 7.5 9 10
}
}
2+3+4
} 5+6 7+8
=9/3 =11/2 =15/2
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 30: 3
7.5
1 10 3
5.5
7.5
S0 = 27 3
n=5 5.5 9
S1 = 28
m=5
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 31: ⎡ n ( n + 1) ⎤
U 0 = nm + ⎢ ⎥ − S0
⎣ 2 ⎦
⎡ 5 ( 5 + 1) ⎤
= 5x5 + ⎢ ⎥ − 27
⎣ 2 ⎦
= 13
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 32: ⎡ n ( n + 1) ⎤
U 0 = nm + ⎢ ⎥ − S0
⎣ 2 ⎦
⎡ 5 ( 5 + 1) ⎤
= 5x5 + ⎢ ⎥ − 27
⎣ 2 ⎦
= 13
90% --> 38
95% --> 41
99% --> 45
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 33: task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 34: Only 2 possible
values: success or fail
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 35: Small samples lead to
very broad estimates
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 36: 4/6 successes =
66.67%
21% - 99.3% with
62.5% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 37: With 30 users
47.7% - 81.9% with
64.8% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 38: s +1
Most likely = p =
n+2
p (1 − p )
Range = p±z
n
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 39: p (1 − p )
p±z
n
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 40: p (1 − p )
p±z
n
most
likely
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 41: p (1 − p )
p±z
n
confidence
level
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 42: p (1 − p )
p±z
n
variability
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 43: A/B
Testing
Photo courtesy of www.dorothyphoto.com
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 44: Compare two different
approaches to the
same problem
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 45: Run both
simultaneously;
randomly divert users
to option B
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 46: Compare using a Chi-
squared test
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 47: Example: clicks on an ad banner
Ignore Click Total
A 10,119 275 10,394
B 962 38 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 48: (e )
2
− oij
χ =∑ 2 ij
eij
The test statistic is a measure of distance
between what we expect to see (e), and what
we actually observed (o). For each cell, subtract
what we expect from what we saw, square it to
remove any negative values, and divide it by
the expected value. Add it all together...
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 49: Calculated expected values
For each cell:
row total x column total/grand
total
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 50: Ignore Click Total
10,108 = 286 =
A 10,394x(11,081/11,394) 10,394x(313/11,394)
10,394
973 = 27 =
B 1,000x(11,081/11,394) 1,000x(313/11,394)
1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 51: Ignore Click Total
A 10,108 - 10,119 = -11 286 - 275 = 11 10,394
B 973 - 962 = 11 27 - 38 = -11 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 52: (e )
2
− oij
χ =∑
2 ij
eij
2 2 2 2
11 11 11 11
= + + +
10,108 286 973 27
= 0.012 + 0.423 + 0.124 + 4.48
= 5.04
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 53: χ 2
α = 0.025 = 5.02 < χ 2
χ 2
α = 0.01 = 6.63 > χ 2
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 54: page views
pre- & post
comparison
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 55: Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 56: Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 57: Can be trending
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 58: Typically compare the
average
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 59: But ignores fluctuation
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 60: But ignores fluctuation
?
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 61: z=
( x1 − x2 )
2 2
s s
+
1 2
n1 n2
2
Test 1 : x1 , s , n1 1
2
Test 2 : x2 , s , n2 2
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 62: z=
( x1 − x2 )
2 2
s s
+
1 2
In order: mean,
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 63: Mean difference
z=
( x1 − x2 )
2 2
s s
+
1 2
In order: mean,
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 64: Mean difference
z=
( x1 − x2 )
2 2
s s
Combined +
1 2
In order: mean,
standard error
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 65: Mean difference
z=
( x1 − x2 )
2 2
s s
Combined +
1 2
In order: mean,
standard error
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
If z < -1.96 or > 1.96 a significance difference exists
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 66: Pre Post
x 1,288 1,331
2 1,369 756.25
s
ni 60 30
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 67: z=
( x1 − x2 )
2 2
s s
+
1 2
n1 n2
=
(1288 − 1331)
1369 756.25
+
60 30
43
= = 6.205
6.93
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 68: 1,288 1,331
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 69: Read more...
Statistics without tears by Derek Rowntree
Flaws & Fallacies in statistical thinking by
Stephen K Campbell
http://uxstats.blogspot.com
Web Directions User Experience ’08 - Analysing Quantitative Data
- Slide 70: Thank you
Web Directions User Experience ’08 - Analysing Quantitative Data