Analysing quantitative
data
with Steve Baty
UX Strategist
Web Directions User Experience ‘08

Data is important
Web Directions User Experience ’08 - Analysing Quantitative Data

We expend
a lot of
effort to
gather it
Web Directions User Experience ’08 - Analysing Quantitative Data

We don’t always use it
well
Web Directions User Experience ’08 - Analysing Quantitative Data

We’ll be looking at:
Web Directions User Experience ’08 - Analysing Quantitative Data

We’ll be looking at:
* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data

We’ll be looking at:
* time-to-completion
* task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data

We’ll be looking at:
* time-to-completion
* task completion rates
* a/b testing
Web Directions User Experience ’08 - Analysing Quantitative Data

We’ll be looking at:
* time-to-completion
* task completion rates
* a/b testing
* page-view data
Web Directions User Experience ’08 - Analysing Quantitative Data

time-to-
completion
Web Directions User Experience ’08 - Analysing Quantitative Data

Web Directions User Experience ’08 - Analysing Quantitative Data

1 min 24 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

1 min 23.8 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

1 min 23.77 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

1 min 23.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

83.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

Our data Task 1
Task 2
User 1
83.5
131.1
User 2 User...
97.3
165.5
might Task 3
Task 4
Task 5
54.5
97.8
118.0
45.5
88.2
143.3
look like Task 6
Task 7
243.9
22.9
309.0
23.9
this...
Web Directions User Experience ’08 - Analysing Quantitative Data

We can calculate...
Web Directions User Experience ’08 - Analysing Quantitative Data

We can calculate...
mean - AVERAGE()
variance - VAR()
standard dev’n - STDEV()
Web Directions User Experience ’08 - Analysing Quantitative Data

Low-variability
Medium-variability
High-variability
90.43 s
Web Directions User Experience ’08 - Analysing Quantitative Data

Compare 2 sets of data
- between iterations
- between audience segments
Web Directions User Experience ’08 - Analysing Quantitative Data

Low sample sizes
restrict options
Web Directions User Experience ’08 - Analysing Quantitative Data

non-parametric
version == no
assumed dist’n
Web Directions User Experience ’08 - Analysing Quantitative Data

Rank-sum test
Web Directions User Experience ’08 - Analysing Quantitative Data

Time for a practical
demonstration
Web Directions User Experience ’08 - Analysing Quantitative Data

Web Directions User Experience ’08 - Analysing Quantitative Data

Web Directions User Experience ’08 - Analysing Quantitative Data

1 3 3 3 5.5 5.5 7.5 7.5 9 10
Web Directions User Experience ’08 - Analysing Quantitative Data

1 3 3 3 5.5 5.5 7.5 7.5 9 10
}
}
2+3+4
} 5+6 7+8
=9/3 =11/2 =15/2
Web Directions User Experience ’08 - Analysing Quantitative Data

3
7.5
1 10 3
5.5
7.5
S0 = 27 3
n=5 5.5 9
S1 = 28
m=5
Web Directions User Experience ’08 - Analysing Quantitative Data

⎡ n ( n + 1) ⎤
U 0 = nm + ⎢ ⎥ − S0
⎣ 2 ⎦
⎡ 5 ( 5 + 1) ⎤
= 5x5 + ⎢ ⎥ − 27
⎣ 2 ⎦
= 13
Web Directions User Experience ’08 - Analysing Quantitative Data

⎡ n ( n + 1) ⎤
U 0 = nm + ⎢ ⎥ − S0
⎣ 2 ⎦
⎡ 5 ( 5 + 1) ⎤
= 5x5 + ⎢ ⎥ − 27
⎣ 2 ⎦
= 13
90% --> 38
95% --> 41
99% --> 45
Web Directions User Experience ’08 - Analysing Quantitative Data

task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data

Only 2 possible
values: success or fail
Web Directions User Experience ’08 - Analysing Quantitative Data

Small samples lead to
very broad estimates
Web Directions User Experience ’08 - Analysing Quantitative Data

4/6 successes =
66.67%
21% - 99.3% with
62.5% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data

With 30 users
47.7% - 81.9% with
64.8% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data

s +1
Most likely = p =
n+2
p (1 − p )
Range = p±z
n
Web Directions User Experience ’08 - Analysing Quantitative Data

p (1 − p )
p±z
n
Web Directions User Experience ’08 - Analysing Quantitative Data

p (1 − p )
p±z
n
most
likely
Web Directions User Experience ’08 - Analysing Quantitative Data

p (1 − p )
p±z
n
confidence
level
Web Directions User Experience ’08 - Analysing Quantitative Data

p (1 − p )
p±z
n
variability
Web Directions User Experience ’08 - Analysing Quantitative Data

A/B
Testing
Photo courtesy of www.dorothyphoto.com
Web Directions User Experience ’08 - Analysing Quantitative Data

Compare two different
approaches to the
same problem
Web Directions User Experience ’08 - Analysing Quantitative Data

Run both
simultaneously;
randomly divert users
to option B
Web Directions User Experience ’08 - Analysing Quantitative Data

Compare using a Chi-
squared test
Web Directions User Experience ’08 - Analysing Quantitative Data

Example: clicks on an ad banner
Ignore Click Total
A 10,119 275 10,394
B 962 38 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data

(e )
2
− oij
χ =∑ 2 ij
eij
The test statistic is a measure of distance
between what we expect to see (e), and what
we actually observed (o). For each cell, subtract
what we expect from what we saw, square it to
remove any negative values, and divide it by
the expected value. Add it all together...
Web Directions User Experience ’08 - Analysing Quantitative Data

Calculated expected values
For each cell:
row total x column total/grand
total
Web Directions User Experience ’08 - Analysing Quantitative Data

Ignore Click Total
10,108 = 286 =
A 10,394x(11,081/11,394) 10,394x(313/11,394)
10,394
973 = 27 =
B 1,000x(11,081/11,394) 1,000x(313/11,394)
1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data

Ignore Click Total
A 10,108 - 10,119 = -11 286 - 275 = 11 10,394
B 973 - 962 = 11 27 - 38 = -11 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data

(e )
2
− oij
χ =∑
2 ij
eij
2 2 2 2
11 11 11 11
= + + +
10,108 286 973 27
= 0.012 + 0.423 + 0.124 + 4.48
= 5.04
Web Directions User Experience ’08 - Analysing Quantitative Data

χ 2
α = 0.025 = 5.02 < χ 2
χ 2
α = 0.01 = 6.63 > χ 2
Web Directions User Experience ’08 - Analysing Quantitative Data

page views
pre- & post
comparison
Web Directions User Experience ’08 - Analysing Quantitative Data

Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data

Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data

Can be trending
Web Directions User Experience ’08 - Analysing Quantitative Data

Typically compare the
average
Web Directions User Experience ’08 - Analysing Quantitative Data

But ignores fluctuation
Web Directions User Experience ’08 - Analysing Quantitative Data

But ignores fluctuation
?
Web Directions User Experience ’08 - Analysing Quantitative Data

z=
( x1 − x2 )
2 2
s s
+
1 2
n1 n2
2
Test 1 : x1 , s , n1 1
2
Test 2 : x2 , s , n2 2
Web Directions User Experience ’08 - Analysing Quantitative Data

z=
( x1 − x2 )
2 2
s s
+
1 2
In order: mean,
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data

Mean difference
z=
( x1 − x2 )
2 2
s s
+
1 2
In order: mean,
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data

Mean difference
z=
( x1 − x2 )
2 2
s s
Combined +
1 2
In order: mean,
standard error
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data

Mean difference
z=
( x1 − x2 )
2 2
s s
Combined +
1 2
In order: mean,
standard error
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
If z < -1.96 or > 1.96 a significance difference exists
Web Directions User Experience ’08 - Analysing Quantitative Data

Pre Post
x 1,288 1,331
2 1,369 756.25
s
ni 60 30
Web Directions User Experience ’08 - Analysing Quantitative Data

z=
( x1 − x2 )
2 2
s s
+
1 2
n1 n2
=
(1288 − 1331)
1369 756.25
+
60 30
43
= = 6.205
6.93
Web Directions User Experience ’08 - Analysing Quantitative Data

1,288 1,331
Web Directions User Experience ’08 - Analysing Quantitative Data

Read more...
Statistics without tears by Derek Rowntree
Flaws & Fallacies in statistical thinking by
Stephen K Campbell
http://uxstats.blogspot.com
Web Directions User Experience ’08 - Analysing Quantitative Data

Thank you
Web Directions User Experience ’08 - Analysing Quantitative Data