•4 likes•613 views

Report

Share

Download to read offline

Are you looking to expand your research toolkit to include some quantitative methods, such as survey research or A/B testing? Have you been asked to collect some usability metrics, but aren’t sure how best to go about that? Or do you just want to be more aware of all of the UX research possibilities? If your answer to any of those questions is yes, then this session is for you. You may know that without statistics, you won’t know if A is really better than B, if users are truly more satisfied with your new site than with your old one, or which changes to your site have actually impacted conversion rates. However, statistics can also help you figure out how to report satisfaction and other metrics you collect during usability tests. And they’re essential for making sense of the results of quantitative usability tests. This session will focus on the statistical concepts that are most useful for UX researchers. It won’t make you a quant, but it will give you a good grounding in quantitative methods and reporting. (For example, you will learn what a margin of error is, how to report quantitative data collected during a usability test - and how not to - and how many people you really need to fill out a survey.)

Follow

- 1. Statistics for UX Professionals Jessica Cameron UX Scotland Thursday, 14 June 2017 1
- 2. Presentation title / Footer text 2Photo by Lane Jackman on Unsplash
- 3. 3 Statistics for UX Professionals Featuring more mean words per slide than any other talk you will see this week @jessscameron
- 4. 4 In the next 60 minutes, you will learn: 1. Useful statistical concepts, like what a confidence interval is and how a margin of error is kind of the same thing as a confidence interval, but not completely 2. How many people you really need to fill out a survey 3. What kinds of quantitative data may be collected during a usability test 4. How to report quantitative data collected during a usability test 5. How not to report quantitative data collected during a usability test
- 5. Presentation title / Footer text 5Photo by Roman Mager on Unsplash
- 6. 6Photo by Carlos Muza on Unsplash Useful statistical concepts
- 7. 7 Why use statistics? • Statistics let us estimate what everyone might do (a population) by looking at what some people do (a sample) • Do not use statistics if you have access to the full population (for example, all members of a project team)
- 8. 8 Key concepts • Central tendency and spread • Mean • Standard deviation • Outlier • Confidence interval • Confidence level • Margin of error
- 9. 9 Central tendency and spread • Central tendency refers to an average, middle, or typical value • Spread refers to how spread out or squeezed together a set of values are (relative to the central tendency) • Example (1 = not at all satisfied, 5 = very satisfied): • Sample 1: 100 people rate a website 1, 100 people rate it 5 • Sample 2: 200 people rate a website 3 • How do the central tendency and spread differ?
- 10. 10 Mean • A mean is a measure of central tendency • The mean is the sum of all measurements divided by the number of measurements • Also known as the average or arithmetic mean • Can be computed in Excel using the AVERAGE function • A sample mean is an estimate of the true population mean
- 11. 11 How much do you like or not like spaghetti? (n=75) 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 12. 12 Mean = 5.87 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 13. 13 Standard deviation • A measure of spread • To find the standard deviation, find the square root of the average of the squared differences between each measurement and the mean • Or use the STDEV function in Excel • If the standard deviation is low, the measurements are grouped close to the mean – and if it is high, they are further apart • If data are normally distributed, 95% of measurements can be expected to fall within 2 standard deviations of the mean
- 15. 15 How much do you like or not like spaghetti? (n=75) 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 16. 16 Mean = 5.87 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 17. 17 1 standard deviation = 3.29 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 18. 18 2 standard deviations = 6.58 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 19. 19 How much do you like or not like spaghetti? 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 20. 20 Mean = 6.85 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 21. 21 1 standard deviation = 2.58 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 22. 22 2 standard deviations = 2.58 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 23. 23 Outlier • Any measurement that falls more than 2 standard deviations from the mean • May also be an unrealistic measurement (for example, time spent on a task that is shorter than the minimum time required by an expert user) • Might represent valid, if extreme, data – such as an important subgroup of users • If 95% of measurements are within 2 standard deviations of the mean, we would expect 5% of the sample to be outliers
- 24. 24 Any outliers here? 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 25. 25 Confidence interval • A measure of spread and sample size • A confidence interval is a range of values that we can say with a certain level of confidence contains the true population mean • This true population mean is the one we would find if we were able to have all possible users perform and/ or rate a task • Use the CONFIDENCE.T function in Excel • (Use alpha = 0.05, which corresponds to a 95% level of confidence)
- 26. 26 95% CI (n=75): 6.26, 7.45 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 27. 27 What determines the size of the confidence interval? • Standard deviation: The larger the standard deviation, the more spread out data are and the larger the confidence interval will be • Sample size: The larger the sample size, the more reliable measurements are and the smaller the confidence interval will be
- 28. 28 95% CI (n=75): 6.26, 7.45 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 29. 29 95% CI (n=375): 6.59, 7.11 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti = 5 people
- 30. 30 What determines the size of the interval? • Standard deviation: The larger the standard deviation, the more spread out data are and the larger the margin of error will be • Sample size: The larger the sample size, the more reliable measurements are and the smaller the confidence interval will be • Confidence level: The higher the level of confidence you need, the larger the confidence interval will have to be
- 31. 31 Confidence level • Confidence intervals are usually reported with a 95% level of confidence • That means that if you repeated a survey 100 times, you would expect the mean response you get to fall within your confidence interval 95 times • You can be 95% confident that the true population mean is within this interval as well • 90% and 99% are also commonly accepted – and the higher the level of confidence, the larger the resulting confidence interval will be
- 32. 32 95% CI (n=375): 6.59, 7.11 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 33. 33 95% CI (n=375): 6.51, 7.20 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 34. 34 Confidence interval – how to report Wrong: • We are 95% confident that the population mean is 6.85 Right: • We can say with 95% confidence that the true population mean lies between 6.51 and 7.20 What is the difference between a confidence interval and a margin of error?
- 35. 35 Margin of error • A margin of error is half of a confidence interval • Excel actually gives us the margin of error, which we then use to compute the confidence interval • A margin of error is usually half of a 95% confidence interval, with the confidence level implied • (That means that it is easier to talk about margin of error than confidence interval) • Margin of error is often used for poll numbers or preferences, and is expressed in percentage points
- 36. 36 Margin of error notes • The ‘error’ refers to random sampling error • It does not account for other errors: • Systematic sampling errors • Errors in survey design (e.g., biased or unclear questions) • Each measurement has a margin of error associated with it • Poll shows Trump at 17% and a bag of hammers at 81%, with a +/- 3 point margin of error • Trump could really get anywhere between 14% and 20% of the vote, and the bag of hammers could get anywhere between 78% and 84% of the vote
- 37. 37 Recap of key concepts • Central tendency and spread • Mean • Standard deviation • Outlier • Confidence interval • Confidence level • Margin of error
- 38. 38 Statistics for surveys Photo by Ranier Ridao on Unsplash
- 39. 39 How many survey participants? (Short answer) 400
- 40. 40 How many survey participants? (Long answer) • It depends on how confident you want to be in the findings (e.g., what size confidence interval are you comfortable with?) • It depends on who your audience is • It depends on the size of the population you are drawing your sample from
- 41. 41 How many participants? (Long answer) • Examples of population sizes: • Everyone in the UK = 60 million • Topshop Oxford Street shoppers = 200,000 per week • Topshop.com shoppers = 4.5 million per week • Topshop employees = 10,000 • A ‘large’ population is anything over about 20,000 • Most surveys of existing (or potential) customers will draw from a large population
- 42. 42 Confidence level = 95% +/- 5 Margin of error +/- 3 +/- 1 Population size 100 80 92 99 500 217 341 475 1,000 278 517 906 10,000 370 965 4,899 20,000 377 1,014 6,489 50,000 382 1,045 8,057 100,000 383 1,056 8,762 500,000 384 1,065 9,423 1,000,000 384 1,066 9,512
- 43. 43 95% CI (n=75): 6.26, 7.45 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 44. 44 95% CI (n=375): 6.59, 7.11 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 45. 45 Collecting vs. comparing data • Surveys typically collect satisfaction data for an existing website at a given time • For that, mean + confidence interval (or margin of error) is an effective way of communicating results • But what if you are comparing two sets of data? • Satisfaction ratings before and after a launch • Conversion rates on versions A and B of a site • Then you will need a proper statistical test
- 46. 46 Use a t-test • To compare two samples, use a two sample t-test • A t-test will tell you if the populations that those samples come from are the same or different (to a certain level of confidence) • It does this by considering the difference between the means relative to the size of the variance, or spread • Not enough to see if confidence levels overlap • That would be an overly conservative test • So instead of eyeballing the data, run a t-test
- 47. 47 How to run a t-test • Use the T.TEST function in Excel – need to know arrays, tails and type • The two arrays are the two data sets • The number of tails depends on what you are predicting • For type, you can generally assume equal variances (type = 2) • Result is a probability that the samples come from the same population • If it is less than .05, you can say that the populations are different
- 48. 48 Mean = 6.85 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 49. 49 Mean = 5.87 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti
- 50. 50 0 1 2 3 4 5 6 7 8 9 10 I hate spaghetti I love spaghetti T-test (N=150): p = .043
- 51. 51Photo by Sandeep Swarnkar on Unsplash
- 52. 52 Rules of thumb • At a 95% confidence level: • A sample size of 400 will yield a margin of error of +/- 5% • A sample size of 1,000 will yield a margin of error of +/- 3% • If you are serious enough about your research to use the recommended number of participants, then you should take your data seriously • Report central tendency and spread • Look for statistically significant results
- 53. Statistics for user testing 53Photo by Rob Hampson on Unsplash
- 54. 54 Data collected during usability tests Qualitative data: • Observations of ease or difficulty participants have completing tasks • What we think about when we think about usability testing Quantitative data: • Success/ failure/ disaster rates (effectiveness) • Task completion time (efficiency) • User surveys (satisfaction)
- 55. 55 Satisfaction data collected during usability tests • Do you collect satisfaction ratings (or any other scale ratings) from usability test participants? • How many people do you usually run during a single usability test? • How do you report those data? • How should you report those data?
- 56. 56 Asking about satisfaction • Questions asked during usability testing can tell us a great deal about the experience that our participants are having on a website • They can tell us basically nothing about the experience that all users have on a website
- 57. 57 Remember confidence intervals? • Your sample size is 6 • Three testers (50%) say they would use the website again • You can say with 95% confidence that the true percentage of the population who would use this website again is somewhere between… • 10% and 90%
- 58. 58 So be careful • Be careful not only with what you choose to ask… • But how you choose to report the data • Graphs and charts can be very compelling… • But so can qualitative data https://finickypenguin.wordpress.com/category/nerd- humor/page/3/
- 59. 59 Takeaways • Reporting confidence intervals along with means gives everyone more confidence in your findings • Try to get 400 people to fill out your surveys • You can use Excel to run simple statistical tests, but if you need something more complicated try R or Stata • Statistics are not the answer to everything… • But if you choose to use them, they can expand your UX toolkit nicely
- 60. Presentation title / Footer text 60Photo by Morvanic Lee on Unsplash Jessica Cameron @jessscameron User Vision @UserVision