Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making Sense of Statistics in HCI: part 3 - gaining power

566 views

Published on

Gaining power – the dreaded ‘too few participants’

http://alandix.com/statistics/course/

Statistical power is about whether an experiment or study is likely to reveal an effect if it is present. Without a sufficiently ‘powerful’ study, you risk being in the middle ground of ‘not proven’, not being able to make a strong statement either for or against whatever effect, system, or theory you are testing.

In HCI studies the greatest problem is often finding sufficient participants to do meaningful statistics. For professional practice we hear that ‘five users are enough’, but less often that this figure was based on particular historical contingencies and in the context of single iterations, not summative evaluations, which still need the equivalent of ‘power’ to be reliable.

However, power arises from a combination of the size of the effect you are trying to detect, the size of the study (number of trails/participants) and the size of the ‘noise’ (the random or uncontrolled factors).

Increasing number of participants is not the only way to increase power and we will discuss various ways in which careful design, selection of subjects and tasks can increase the power of your study albeit sometimes requiring care in interpreting results. For example, using a very narrow user group can reduce individual differences in knowledge and skill (reduce noise) and make it easier to see the effect of a novel interaction technique, but also reduces generalisation beyond that group. In another example, we will also see how careful choice of a task can even be used to deal with infrequent expert slips.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Making Sense of Statistics in HCI: part 3 - gaining power

  1. 1. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Making Sense of Statistics in HCI Part 3 – Gaining Power the dreaded ‘too few participants’ Alan Dix http://alandix.com/statistics/
  2. 2. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix don’t you hate it when …
  3. 3. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix statistical power – what is it? if there is a real effect how likely are you to be able to detect it? avoiding false negatives
  4. 4. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix increasing power standard approach … add more participants but not the only way! can get more power … but often sacrifice a little generality need to understand and explain with great power comes great responsibility ;-)
  5. 5. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  6. 6. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix noise–effect–number triangle three routes to gain power
  7. 7. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix recall Stats 101 (for simple data) σ – standard deviation of data often ‘noise’ – things you can’t control or measure e.g. individual variability s.e. – standard error of mean (s.e.) the accuracy of your estimate (error bars) s.e. = σ / √n (if σ is an estimate √n-1 ) to half standard error you must quadruple number.
  8. 8. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix effect size how big a difference do you want to detect? call it δ the accuracy (s.e.) needs to be better than δ δ >> σ / √n
  9. 9. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix δ >> σ / √n effect size noise number of participants
  10. 10. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix noise–effect–number triangle to gain power address any of these not just more subjects!
  11. 11. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix general strategies increase number – the standard approach … but … – square root often means very large increases reduce noise – control conditions (physics approach) – measure other factors and fit (e.g. age, experience) increase effect size – manipulate sensitivity (e.g. photo back of crowd!)
  12. 12. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  13. 13. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix subjects control or manipulate who
  14. 14. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix more subjects or trials more subjects – average out between subject differences more trials – average out within subject variation e.g. Fitts’ Law experiments … but both both may need lots e.g. to reduce noise by 10, need 100 times more
  15. 15. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix within subject/group cancels out between subject variation helpful if effect reasonably consistent but between subject variability high may cause problems with order effects, learning condition A condition B Subjcet # 1 2 3 4 5 6 7 8 9 10
  16. 16. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix narrow/matched users aims to reduce between subject variation choose subjects who are very similar to each other or in some way have matched e.g. balance gender, skills allows between subject experiments how do you know what is important to match?
  17. 17. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix targeted user group aims to increase the effect size choose group of users who are likely to be especially affected by the intervention e.g. novices or older users but … generalisation to other users will be by theoretical argument not empirical data
  18. 18. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  19. 19. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix tasks control or manipulate what
  20. 20. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix distractor tasks aim to saturate user’s cognitive resources so make them more sensitive to intervention e.g. count backwards while performing task helpful when coping mechanisms mask effects condition A B mental load capacity condition A B overload leads to errors add distractor
  21. 21. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix targeted tasks design a task that will expose effect of intervention e.g. trouble with buttons paper (expert slip) … but … care again with generalisation!
  22. 22. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix example: trouble with buttons error-free behaviour release mouse over button press mouse over button expert slip move off button first … and then release
  23. 23. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix trouble with buttons (2) novices: work more slowly – less likely to make slip notice lack of semantic feedback – so they recover experts: act quickly – so make more slips focused on next action, so miss feedback problem: experts slips don’t happen often … never in experiments needed to craft task to engineer expert slips
  24. 24. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix demonic interventions! extreme version create deliberately nasty task! e.g. 'natural inverse' steering task Fitts’ Law-ish experiment added artificial errors to cause overshoots … but … again generalisation … and … subjects may hate you!
  25. 25. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix reduced vs. wild in the wild has lots of extraneous effects = noise! control environment => lab or semi-wild reduced task e.g. scripted use in wild environment reduced system e.g. mobile tourist app with less options
  26. 26. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix

×