Slideshow Transcript

1. Slide 1: Analysing quantitative data with Steve Baty UX Strategist Web Directions User Experience ‘08
2. Slide 2: Data is important Web Directions User Experience ’08 - Analysing Quantitative Data
3. Slide 3: We expend a lot of effort to gather it Web Directions User Experience ’08 - Analysing Quantitative Data
4. Slide 4: We don’t always use it well Web Directions User Experience ’08 - Analysing Quantitative Data
5. Slide 5: We’ll be looking at: Web Directions User Experience ’08 - Analysing Quantitative Data
6. Slide 6: We’ll be looking at: * time-to-completion Web Directions User Experience ’08 - Analysing Quantitative Data
7. Slide 7: We’ll be looking at: * time-to-completion * task completion rates Web Directions User Experience ’08 - Analysing Quantitative Data
8. Slide 8: We’ll be looking at: * time-to-completion * task completion rates * a/b testing Web Directions User Experience ’08 - Analysing Quantitative Data
9. Slide 9: We’ll be looking at: * time-to-completion * task completion rates * a/b testing * page-view data Web Directions User Experience ’08 - Analysing Quantitative Data
10. Slide 10: time-to- completion Web Directions User Experience ’08 - Analysing Quantitative Data
11. Slide 11: Web Directions User Experience ’08 - Analysing Quantitative Data
12. Slide 12: 1 min 24 secs Web Directions User Experience ’08 - Analysing Quantitative Data
13. Slide 13: 1 min 23.8 secs Web Directions User Experience ’08 - Analysing Quantitative Data
14. Slide 14: 1 min 23.77 secs Web Directions User Experience ’08 - Analysing Quantitative Data
15. Slide 15: 1 min 23.768 secs Web Directions User Experience ’08 - Analysing Quantitative Data
16. Slide 16: 83.768 secs Web Directions User Experience ’08 - Analysing Quantitative Data
17. Slide 17: Our data Task 1 Task 2 User 1 83.5 131.1 User 2 User... 97.3 165.5 might Task 3 Task 4 Task 5 54.5 97.8 118.0 45.5 88.2 143.3 look like Task 6 Task 7 243.9 22.9 309.0 23.9 this... Web Directions User Experience ’08 - Analysing Quantitative Data
18. Slide 18: We can calculate... Web Directions User Experience ’08 - Analysing Quantitative Data
19. Slide 19: We can calculate... mean - AVERAGE() variance - VAR() standard dev’n - STDEV() Web Directions User Experience ’08 - Analysing Quantitative Data
20. Slide 20: Low-variability Medium-variability High-variability 90.43 s Web Directions User Experience ’08 - Analysing Quantitative Data
21. Slide 21: Compare 2 sets of data - between iterations - between audience segments Web Directions User Experience ’08 - Analysing Quantitative Data
22. Slide 22: Low sample sizes restrict options Web Directions User Experience ’08 - Analysing Quantitative Data
23. Slide 23: non-parametric version == no assumed dist’n Web Directions User Experience ’08 - Analysing Quantitative Data
24. Slide 24: Rank-sum test Web Directions User Experience ’08 - Analysing Quantitative Data
25. Slide 25: Time for a practical demonstration Web Directions User Experience ’08 - Analysing Quantitative Data
26. Slide 26: Web Directions User Experience ’08 - Analysing Quantitative Data
27. Slide 27: Web Directions User Experience ’08 - Analysing Quantitative Data
28. Slide 28: 1 3 3 3 5.5 5.5 7.5 7.5 9 10 Web Directions User Experience ’08 - Analysing Quantitative Data
29. Slide 29: 1 3 3 3 5.5 5.5 7.5 7.5 9 10 } } 2+3+4 } 5+6 7+8 =9/3 =11/2 =15/2 Web Directions User Experience ’08 - Analysing Quantitative Data
30. Slide 30: 3 7.5 1 10 3 5.5 7.5 S0 = 27 3 n=5 5.5 9 S1 = 28 m=5 Web Directions User Experience ’08 - Analysing Quantitative Data
31. Slide 31: ⎡ n ( n + 1) ⎤ U 0 = nm + ⎢ ⎥ − S0 ⎣ 2 ⎦ ⎡ 5 ( 5 + 1) ⎤ = 5x5 + ⎢ ⎥ − 27 ⎣ 2 ⎦ = 13 Web Directions User Experience ’08 - Analysing Quantitative Data
32. Slide 32: ⎡ n ( n + 1) ⎤ U 0 = nm + ⎢ ⎥ − S0 ⎣ 2 ⎦ ⎡ 5 ( 5 + 1) ⎤ = 5x5 + ⎢ ⎥ − 27 ⎣ 2 ⎦ = 13 90% --> 38 95% --> 41 99% --> 45 Web Directions User Experience ’08 - Analysing Quantitative Data
33. Slide 33: task completion rates Web Directions User Experience ’08 - Analysing Quantitative Data
34. Slide 34: Only 2 possible values: success or fail Web Directions User Experience ’08 - Analysing Quantitative Data
35. Slide 35: Small samples lead to very broad estimates Web Directions User Experience ’08 - Analysing Quantitative Data
36. Slide 36: 4/6 successes = 66.67% 21% - 99.3% with 62.5% most likely Web Directions User Experience ’08 - Analysing Quantitative Data
37. Slide 37: With 30 users 47.7% - 81.9% with 64.8% most likely Web Directions User Experience ’08 - Analysing Quantitative Data
38. Slide 38: s +1 Most likely = p = n+2 p (1 − p ) Range = p±z n Web Directions User Experience ’08 - Analysing Quantitative Data
39. Slide 39: p (1 − p ) p±z n Web Directions User Experience ’08 - Analysing Quantitative Data
40. Slide 40: p (1 − p ) p±z n most likely Web Directions User Experience ’08 - Analysing Quantitative Data
41. Slide 41: p (1 − p ) p±z n confidence level Web Directions User Experience ’08 - Analysing Quantitative Data
42. Slide 42: p (1 − p ) p±z n variability Web Directions User Experience ’08 - Analysing Quantitative Data
43. Slide 43: A/B Testing Photo courtesy of www.dorothyphoto.com Web Directions User Experience ’08 - Analysing Quantitative Data
44. Slide 44: Compare two different approaches to the same problem Web Directions User Experience ’08 - Analysing Quantitative Data
45. Slide 45: Run both simultaneously; randomly divert users to option B Web Directions User Experience ’08 - Analysing Quantitative Data
46. Slide 46: Compare using a Chi- squared test Web Directions User Experience ’08 - Analysing Quantitative Data
47. Slide 47: Example: clicks on an ad banner Ignore Click Total A 10,119 275 10,394 B 962 38 1,000 Total 11,081 313 11,394 Web Directions User Experience ’08 - Analysing Quantitative Data
48. Slide 48: (e ) 2 − oij χ =∑ 2 ij eij The test statistic is a measure of distance between what we expect to see (e), and what we actually observed (o). For each cell, subtract what we expect from what we saw, square it to remove any negative values, and divide it by the expected value. Add it all together... Web Directions User Experience ’08 - Analysing Quantitative Data
49. Slide 49: Calculated expected values For each cell: row total x column total/grand total Web Directions User Experience ’08 - Analysing Quantitative Data
50. Slide 50: Ignore Click Total 10,108 = 286 = A 10,394x(11,081/11,394) 10,394x(313/11,394) 10,394 973 = 27 = B 1,000x(11,081/11,394) 1,000x(313/11,394) 1,000 Total 11,081 313 11,394 Web Directions User Experience ’08 - Analysing Quantitative Data
51. Slide 51: Ignore Click Total A 10,108 - 10,119 = -11 286 - 275 = 11 10,394 B 973 - 962 = 11 27 - 38 = -11 1,000 Total 11,081 313 11,394 Web Directions User Experience ’08 - Analysing Quantitative Data
52. Slide 52: (e ) 2 − oij χ =∑ 2 ij eij 2 2 2 2 11 11 11 11 = + + + 10,108 286 973 27 = 0.012 + 0.423 + 0.124 + 4.48 = 5.04 Web Directions User Experience ’08 - Analysing Quantitative Data
53. Slide 53: χ 2 α = 0.025 = 5.02 < χ 2 χ 2 α = 0.01 = 6.63 > χ 2 Web Directions User Experience ’08 - Analysing Quantitative Data
54. Slide 54: page views pre- & post comparison Web Directions User Experience ’08 - Analysing Quantitative Data
55. Slide 55: Can be cyclical Web Directions User Experience ’08 - Analysing Quantitative Data
56. Slide 56: Can be cyclical Web Directions User Experience ’08 - Analysing Quantitative Data
57. Slide 57: Can be trending Web Directions User Experience ’08 - Analysing Quantitative Data
58. Slide 58: Typically compare the average Web Directions User Experience ’08 - Analysing Quantitative Data
59. Slide 59: But ignores fluctuation Web Directions User Experience ’08 - Analysing Quantitative Data
60. Slide 60: But ignores fluctuation ? Web Directions User Experience ’08 - Analysing Quantitative Data
61. Slide 61: z= ( x1 − x2 ) 2 2 s s + 1 2 n1 n2 2 Test  1 : x1 , s , n1 1 2 Test  2 : x2 , s , n2 2 Web Directions User Experience ’08 - Analysing Quantitative Data
62. Slide 62: z= ( x1 − x2 ) 2 2 s s + 1 2 In order: mean, n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. Web Directions User Experience ’08 - Analysing Quantitative Data
63. Slide 63: Mean difference z= ( x1 − x2 ) 2 2 s s + 1 2 In order: mean, n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. Web Directions User Experience ’08 - Analysing Quantitative Data
64. Slide 64: Mean difference z= ( x1 − x2 ) 2 2 s s Combined + 1 2 In order: mean, standard error n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. Web Directions User Experience ’08 - Analysing Quantitative Data
65. Slide 65: Mean difference z= ( x1 − x2 ) 2 2 s s Combined + 1 2 In order: mean, standard error n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. If z < -1.96 or > 1.96 a significance difference exists Web Directions User Experience ’08 - Analysing Quantitative Data
66. Slide 66: Pre Post x 1,288 1,331 2 1,369 756.25 s ni 60 30 Web Directions User Experience ’08 - Analysing Quantitative Data
67. Slide 67: z= ( x1 − x2 ) 2 2 s s + 1 2 n1 n2 = (1288 − 1331) 1369 756.25 + 60 30 43 = = 6.205 6.93 Web Directions User Experience ’08 - Analysing Quantitative Data
68. Slide 68: 1,288 1,331 Web Directions User Experience ’08 - Analysing Quantitative Data
69. Slide 69: Read more... Statistics without tears by Derek Rowntree Flaws & Fallacies in statistical thinking by Stephen K Campbell http://uxstats.blogspot.com Web Directions User Experience ’08 - Analysing Quantitative Data
70. Slide 70: Thank you Web Directions User Experience ’08 - Analysing Quantitative Data