Analysing quantitative data

5,241 views

Published on

An introduction to the analysis of quantitative data arising from user research

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,241
On SlideShare
0
From Embeds
0
Number of Embeds
265
Actions
Shares
0
Downloads
242
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Analysing quantitative data

  1. 1. Analysing quantitative data with Steve Baty UX Strategist Web Directions User Experience ‘08
  2. 2. Data is important Web Directions User Experience ’08 - Analysing Quantitative Data
  3. 3. We expend a lot of effort to gather it Web Directions User Experience ’08 - Analysing Quantitative Data
  4. 4. We don’t always use it well Web Directions User Experience ’08 - Analysing Quantitative Data
  5. 5. We’ll be looking at: Web Directions User Experience ’08 - Analysing Quantitative Data
  6. 6. We’ll be looking at: * time-to-completion Web Directions User Experience ’08 - Analysing Quantitative Data
  7. 7. We’ll be looking at: * time-to-completion * task completion rates Web Directions User Experience ’08 - Analysing Quantitative Data
  8. 8. We’ll be looking at: * time-to-completion * task completion rates * a/b testing Web Directions User Experience ’08 - Analysing Quantitative Data
  9. 9. We’ll be looking at: * time-to-completion * task completion rates * a/b testing * page-view data Web Directions User Experience ’08 - Analysing Quantitative Data
  10. 10. time-to- completion Web Directions User Experience ’08 - Analysing Quantitative Data
  11. 11. Web Directions User Experience ’08 - Analysing Quantitative Data
  12. 12. 1 min 24 secs Web Directions User Experience ’08 - Analysing Quantitative Data
  13. 13. 1 min 23.8 secs Web Directions User Experience ’08 - Analysing Quantitative Data
  14. 14. 1 min 23.77 secs Web Directions User Experience ’08 - Analysing Quantitative Data
  15. 15. 1 min 23.768 secs Web Directions User Experience ’08 - Analysing Quantitative Data
  16. 16. 83.768 secs Web Directions User Experience ’08 - Analysing Quantitative Data
  17. 17. Our data Task 1 Task 2 User 1 83.5 131.1 User 2 User... 97.3 165.5 might Task 3 Task 4 Task 5 54.5 97.8 118.0 45.5 88.2 143.3 look like Task 6 Task 7 243.9 22.9 309.0 23.9 this... Web Directions User Experience ’08 - Analysing Quantitative Data
  18. 18. We can calculate... Web Directions User Experience ’08 - Analysing Quantitative Data
  19. 19. We can calculate... mean - AVERAGE() variance - VAR() standard dev’n - STDEV() Web Directions User Experience ’08 - Analysing Quantitative Data
  20. 20. Low-variability Medium-variability High-variability 90.43 s Web Directions User Experience ’08 - Analysing Quantitative Data
  21. 21. Compare 2 sets of data - between iterations - between audience segments Web Directions User Experience ’08 - Analysing Quantitative Data
  22. 22. Low sample sizes restrict options Web Directions User Experience ’08 - Analysing Quantitative Data
  23. 23. non-parametric version == no assumed dist’n Web Directions User Experience ’08 - Analysing Quantitative Data
  24. 24. Rank-sum test Web Directions User Experience ’08 - Analysing Quantitative Data
  25. 25. Time for a practical demonstration Web Directions User Experience ’08 - Analysing Quantitative Data
  26. 26. Web Directions User Experience ’08 - Analysing Quantitative Data
  27. 27. Web Directions User Experience ’08 - Analysing Quantitative Data
  28. 28. 1 3 3 3 5.5 5.5 7.5 7.5 9 10 Web Directions User Experience ’08 - Analysing Quantitative Data
  29. 29. 1 3 3 3 5.5 5.5 7.5 7.5 9 10 } } 2+3+4 } 5+6 7+8 =9/3 =11/2 =15/2 Web Directions User Experience ’08 - Analysing Quantitative Data
  30. 30. 3 7.5 1 10 3 5.5 7.5 S0 = 27 3 n=5 5.5 9 S1 = 28 m=5 Web Directions User Experience ’08 - Analysing Quantitative Data
  31. 31. ⎡ n ( n + 1) ⎤ U 0 = nm + ⎢ ⎥ − S0 ⎣ 2 ⎦ ⎡ 5 ( 5 + 1) ⎤ = 5x5 + ⎢ ⎥ − 27 ⎣ 2 ⎦ = 13 Web Directions User Experience ’08 - Analysing Quantitative Data
  32. 32. ⎡ n ( n + 1) ⎤ U 0 = nm + ⎢ ⎥ − S0 ⎣ 2 ⎦ ⎡ 5 ( 5 + 1) ⎤ = 5x5 + ⎢ ⎥ − 27 ⎣ 2 ⎦ = 13 90% --> 38 95% --> 41 99% --> 45 Web Directions User Experience ’08 - Analysing Quantitative Data
  33. 33. task completion rates Web Directions User Experience ’08 - Analysing Quantitative Data
  34. 34. Only 2 possible values: success or fail Web Directions User Experience ’08 - Analysing Quantitative Data
  35. 35. Small samples lead to very broad estimates Web Directions User Experience ’08 - Analysing Quantitative Data
  36. 36. 4/6 successes = 66.67% 21% - 99.3% with 62.5% most likely Web Directions User Experience ’08 - Analysing Quantitative Data
  37. 37. With 30 users 47.7% - 81.9% with 64.8% most likely Web Directions User Experience ’08 - Analysing Quantitative Data
  38. 38. s +1 Most likely = p = n+2 p (1 − p ) Range = p±z n Web Directions User Experience ’08 - Analysing Quantitative Data
  39. 39. p (1 − p ) p±z n Web Directions User Experience ’08 - Analysing Quantitative Data
  40. 40. p (1 − p ) p±z n most likely Web Directions User Experience ’08 - Analysing Quantitative Data
  41. 41. p (1 − p ) p±z n confidence level Web Directions User Experience ’08 - Analysing Quantitative Data
  42. 42. p (1 − p ) p±z n variability Web Directions User Experience ’08 - Analysing Quantitative Data
  43. 43. A/B Testing Photo courtesy of www.dorothyphoto.com Web Directions User Experience ’08 - Analysing Quantitative Data
  44. 44. Compare two different approaches to the same problem Web Directions User Experience ’08 - Analysing Quantitative Data
  45. 45. Run both simultaneously; randomly divert users to option B Web Directions User Experience ’08 - Analysing Quantitative Data
  46. 46. Compare using a Chi- squared test Web Directions User Experience ’08 - Analysing Quantitative Data
  47. 47. Example: clicks on an ad banner Ignore Click Total A 10,119 275 10,394 B 962 38 1,000 Total 11,081 313 11,394 Web Directions User Experience ’08 - Analysing Quantitative Data
  48. 48. (e ) 2 − oij χ =∑ 2 ij eij The test statistic is a measure of distance between what we expect to see (e), and what we actually observed (o). For each cell, subtract what we expect from what we saw, square it to remove any negative values, and divide it by the expected value. Add it all together... Web Directions User Experience ’08 - Analysing Quantitative Data
  49. 49. Calculated expected values For each cell: row total x column total/grand total Web Directions User Experience ’08 - Analysing Quantitative Data
  50. 50. Ignore Click Total 10,108 = 286 = A 10,394x(11,081/11,394) 10,394x(313/11,394) 10,394 973 = 27 = B 1,000x(11,081/11,394) 1,000x(313/11,394) 1,000 Total 11,081 313 11,394 Web Directions User Experience ’08 - Analysing Quantitative Data
  51. 51. Ignore Click Total A 10,108 - 10,119 = -11 286 - 275 = 11 10,394 B 973 - 962 = 11 27 - 38 = -11 1,000 Total 11,081 313 11,394 Web Directions User Experience ’08 - Analysing Quantitative Data
  52. 52. (e ) 2 − oij χ =∑ 2 ij eij 2 2 2 2 11 11 11 11 = + + + 10,108 286 973 27 = 0.012 + 0.423 + 0.124 + 4.48 = 5.04 Web Directions User Experience ’08 - Analysing Quantitative Data
  53. 53. χ 2 α = 0.025 = 5.02 < χ 2 χ 2 α = 0.01 = 6.63 > χ 2 Web Directions User Experience ’08 - Analysing Quantitative Data
  54. 54. page views pre- & post comparison Web Directions User Experience ’08 - Analysing Quantitative Data
  55. 55. Can be cyclical Web Directions User Experience ’08 - Analysing Quantitative Data
  56. 56. Can be cyclical Web Directions User Experience ’08 - Analysing Quantitative Data
  57. 57. Can be trending Web Directions User Experience ’08 - Analysing Quantitative Data
  58. 58. Typically compare the average Web Directions User Experience ’08 - Analysing Quantitative Data
  59. 59. But ignores fluctuation Web Directions User Experience ’08 - Analysing Quantitative Data
  60. 60. But ignores fluctuation ? Web Directions User Experience ’08 - Analysing Quantitative Data
  61. 61. z= ( x1 − x2 ) 2 2 s s + 1 2 n1 n2 2 Test  1 : x1 , s , n1 1 2 Test  2 : x2 , s , n2 2 Web Directions User Experience ’08 - Analysing Quantitative Data
  62. 62. z= ( x1 − x2 ) 2 2 s s + 1 2 In order: mean, n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. Web Directions User Experience ’08 - Analysing Quantitative Data
  63. 63. Mean difference z= ( x1 − x2 ) 2 2 s s + 1 2 In order: mean, n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. Web Directions User Experience ’08 - Analysing Quantitative Data
  64. 64. Mean difference z= ( x1 − x2 ) 2 2 s s Combined + 1 2 In order: mean, standard error n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. Web Directions User Experience ’08 - Analysing Quantitative Data
  65. 65. Mean difference z= ( x1 − x2 ) 2 2 s s Combined + 1 2 In order: mean, standard error n1 n2 variance & Test  1 : x1 , s , n1 2 number of data 1 points in each 2 Test  2 : x2 , s , n2 2 test. If z < -1.96 or > 1.96 a significance difference exists Web Directions User Experience ’08 - Analysing Quantitative Data
  66. 66. Pre Post x 1,288 1,331 2 1,369 756.25 s ni 60 30 Web Directions User Experience ’08 - Analysing Quantitative Data
  67. 67. z= ( x1 − x2 ) 2 2 s s + 1 2 n1 n2 = (1288 − 1331) 1369 756.25 + 60 30 43 = = 6.205 6.93 Web Directions User Experience ’08 - Analysing Quantitative Data
  68. 68. 1,288 1,331 Web Directions User Experience ’08 - Analysing Quantitative Data
  69. 69. Read more... Statistics without tears by Derek Rowntree Flaws & Fallacies in statistical thinking by Stephen K Campbell http://uxstats.blogspot.com Web Directions User Experience ’08 - Analysing Quantitative Data
  70. 70. Thank you Web Directions User Experience ’08 - Analysing Quantitative Data

×