The document discusses analyzing quantitative user experience data. It will cover topics like time-to-completion, task completion rates, A/B testing, and page-view data. Examples are provided of what quantitative data may look like, including times for users to complete tasks. Methods for calculating averages, variance and standard deviation of data are also outlined. A/B testing is discussed as a way to compare two approaches and measure differences in metrics like click-through rates.

1. Analysing quantitative
data
with Steve Baty
UX Strategist
Web Directions User Experience ‘08

2. Data is important
Web Directions User Experience ’08 - Analysing Quantitative Data

3. We expend
a lot of
effort to
gather it
Web Directions User Experience ’08 - Analysing Quantitative Data

4. We don’t always use it
well
Web Directions User Experience ’08 - Analysing Quantitative Data

5. We’ll be looking at:
Web Directions User Experience ’08 - Analysing Quantitative Data

6. We’ll be looking at:
* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data

7. We’ll be looking at:
* time-to-completion
* task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data

8. We’ll be looking at:
* time-to-completion
* task completion rates
* a/b testing
Web Directions User Experience ’08 - Analysing Quantitative Data

9. We’ll be looking at:
* time-to-completion
* task completion rates
* a/b testing
* page-view data
Web Directions User Experience ’08 - Analysing Quantitative Data

10. time-to-
completion
Web Directions User Experience ’08 - Analysing Quantitative Data

11. Web Directions User Experience ’08 - Analysing Quantitative Data

12. 1 min 24 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

13. 1 min 23.8 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

14. 1 min 23.77 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

15. 1 min 23.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

16. 83.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data

17. Our data Task 1
Task 2
User 1
83.5
131.1
User 2 User...
97.3
165.5
might Task 3
Task 4
Task 5
54.5
97.8
118.0
45.5
88.2
143.3
look like Task 6
Task 7
243.9
22.9
309.0
23.9
this...
Web Directions User Experience ’08 - Analysing Quantitative Data

18. We can calculate...
Web Directions User Experience ’08 - Analysing Quantitative Data

19. We can calculate...
mean - AVERAGE()
variance - VAR()
standard dev’n - STDEV()
Web Directions User Experience ’08 - Analysing Quantitative Data

20. Low-variability
Medium-variability
High-variability
90.43 s
Web Directions User Experience ’08 - Analysing Quantitative Data

21. Compare 2 sets of data
- between iterations
- between audience segments
Web Directions User Experience ’08 - Analysing Quantitative Data

22. Low sample sizes
restrict options
Web Directions User Experience ’08 - Analysing Quantitative Data

23. non-parametric
version == no
assumed dist’n
Web Directions User Experience ’08 - Analysing Quantitative Data

24. Rank-sum test
Web Directions User Experience ’08 - Analysing Quantitative Data

25. Time for a practical
demonstration
Web Directions User Experience ’08 - Analysing Quantitative Data

26. Web Directions User Experience ’08 - Analysing Quantitative Data

27. Web Directions User Experience ’08 - Analysing Quantitative Data

28. 1 3 3 3 5.5 5.5 7.5 7.5 9 10
Web Directions User Experience ’08 - Analysing Quantitative Data

29. 1 3 3 3 5.5 5.5 7.5 7.5 9 10
}
}
2+3+4
} 5+6 7+8
=9/3 =11/2 =15/2
Web Directions User Experience ’08 - Analysing Quantitative Data

30. 3
7.5
1 10 3
5.5
7.5
S0 = 27 3
n=5 5.5 9
S1 = 28
m=5
Web Directions User Experience ’08 - Analysing Quantitative Data

31. ⎡ n ( n + 1) ⎤
U 0 = nm + ⎢ ⎥ − S0
⎣ 2 ⎦
⎡ 5 ( 5 + 1) ⎤
= 5x5 + ⎢ ⎥ − 27
⎣ 2 ⎦
= 13
Web Directions User Experience ’08 - Analysing Quantitative Data

32. ⎡ n ( n + 1) ⎤
U 0 = nm + ⎢ ⎥ − S0
⎣ 2 ⎦
⎡ 5 ( 5 + 1) ⎤
= 5x5 + ⎢ ⎥ − 27
⎣ 2 ⎦
= 13
90% --> 38
95% --> 41
99% --> 45
Web Directions User Experience ’08 - Analysing Quantitative Data

33. task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data

34. Only 2 possible
values: success or fail
Web Directions User Experience ’08 - Analysing Quantitative Data

35. Small samples lead to
very broad estimates
Web Directions User Experience ’08 - Analysing Quantitative Data

36. 4/6 successes =
66.67%
21% - 99.3% with
62.5% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data

37. With 30 users
47.7% - 81.9% with
64.8% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data

38. s +1
Most likely = p =
n+2
p (1 − p )
Range = p±z
n
Web Directions User Experience ’08 - Analysing Quantitative Data

39. p (1 − p )
p±z
n
Web Directions User Experience ’08 - Analysing Quantitative Data

40. p (1 − p )
p±z
n
most
likely
Web Directions User Experience ’08 - Analysing Quantitative Data

41. p (1 − p )
p±z
n
confidence
level
Web Directions User Experience ’08 - Analysing Quantitative Data

42. p (1 − p )
p±z
n
variability
Web Directions User Experience ’08 - Analysing Quantitative Data

43. A/B
Testing
Photo courtesy of www.dorothyphoto.com
Web Directions User Experience ’08 - Analysing Quantitative Data

44. Compare two different
approaches to the
same problem
Web Directions User Experience ’08 - Analysing Quantitative Data

45. Run both
simultaneously;
randomly divert users
to option B
Web Directions User Experience ’08 - Analysing Quantitative Data

46. Compare using a Chi-
squared test
Web Directions User Experience ’08 - Analysing Quantitative Data

47. Example: clicks on an ad banner
Ignore Click Total
A 10,119 275 10,394
B 962 38 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data

48. (e )
2
− oij
χ =∑ 2 ij
eij
The test statistic is a measure of distance
between what we expect to see (e), and what
we actually observed (o). For each cell, subtract
what we expect from what we saw, square it to
remove any negative values, and divide it by
the expected value. Add it all together...
Web Directions User Experience ’08 - Analysing Quantitative Data

49. Calculated expected values
For each cell:
row total x column total/grand
total
Web Directions User Experience ’08 - Analysing Quantitative Data

50. Ignore Click Total
10,108 = 286 =
A 10,394x(11,081/11,394) 10,394x(313/11,394)
10,394
973 = 27 =
B 1,000x(11,081/11,394) 1,000x(313/11,394)
1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data

51. Ignore Click Total
A 10,108 - 10,119 = -11 286 - 275 = 11 10,394
B 973 - 962 = 11 27 - 38 = -11 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data

52. (e )
2
− oij
χ =∑
2 ij
eij
2 2 2 2
11 11 11 11
= + + +
10,108 286 973 27
= 0.012 + 0.423 + 0.124 + 4.48
= 5.04
Web Directions User Experience ’08 - Analysing Quantitative Data

53. χ 2
α = 0.025 = 5.02 < χ 2
χ 2
α = 0.01 = 6.63 > χ 2
Web Directions User Experience ’08 - Analysing Quantitative Data

54. page views
pre- & post
comparison
Web Directions User Experience ’08 - Analysing Quantitative Data

55. Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data

56. Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data

57. Can be trending
Web Directions User Experience ’08 - Analysing Quantitative Data

58. Typically compare the
average
Web Directions User Experience ’08 - Analysing Quantitative Data

59. But ignores fluctuation
Web Directions User Experience ’08 - Analysing Quantitative Data

60. But ignores fluctuation
?
Web Directions User Experience ’08 - Analysing Quantitative Data

61. z=
( x1 − x2 )
2 2
s s
+
1 2
n1 n2
2
Test 1 : x1 , s , n1 1
2
Test 2 : x2 , s , n2 2
Web Directions User Experience ’08 - Analysing Quantitative Data

62. z=
( x1 − x2 )
2 2
s s
+
1 2
In order: mean,
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data

63. Mean difference
z=
( x1 − x2 )
2 2
s s
+
1 2
In order: mean,
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data

64. Mean difference
z=
( x1 − x2 )
2 2
s s
Combined +
1 2
In order: mean,
standard error
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
Web Directions User Experience ’08 - Analysing Quantitative Data

65. Mean difference
z=
( x1 − x2 )
2 2
s s
Combined +
1 2
In order: mean,
standard error
n1 n2
variance &
Test 1 : x1 , s , n1 2 number of data
1
points in each
2
Test 2 : x2 , s , n2 2 test.
If z < -1.96 or > 1.96 a significance difference exists
Web Directions User Experience ’08 - Analysing Quantitative Data

66. Pre Post
x 1,288 1,331
2 1,369 756.25
s
ni 60 30
Web Directions User Experience ’08 - Analysing Quantitative Data

67. z=
( x1 − x2 )
2 2
s s
+
1 2
n1 n2
=
(1288 − 1331)
1369 756.25
+
60 30
43
= = 6.205
6.93
Web Directions User Experience ’08 - Analysing Quantitative Data

68. 1,288 1,331
Web Directions User Experience ’08 - Analysing Quantitative Data

69. Read more...
Statistics without tears by Derek Rowntree
Flaws & Fallacies in statistical thinking by
Stephen K Campbell
http://uxstats.blogspot.com
Web Directions User Experience ’08 - Analysing Quantitative Data

70. Thank you
Web Directions User Experience ’08 - Analysing Quantitative Data