Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Making Sense of Statistics in HCI: part 1 - wild and wide

658 views

Published on

Wild and wide – concerning randomness and distributions

http://alandix.com/statistics/course/

The course will begin with some exercises and demonstrations of the unexpected wildness of random phenomena including the effects of bias and non-independence (when one result effects others).

We will discuss different kinds of distribution and the reasons why the normal distribution (classic hat shape), on which so many statistical tests are based, is so common. In particular we will look at some of the ways in which the effects we see in HCI may not satisfy the assumptions behind the normal distribution.

Most will be aware of the use of non-parametric statistics for discrete data such as Likert scales, but there are other ways in which non-normal distributions arise. Positive feedback effects, which give rise to the beauty of a snowflake, also create effects such as the bi-modal distribution of student marks in certain kinds of university courses (don’t believe those who say marks should be normally distributed!). This can become more complex if feedback processes include some form of threshold or other non-linear effect (e.g. when the rate of a task just gets too much for a user).

All of these effects are found in the processes that give rise to social networks both online and offline and other forms of network phenomena, which are often far better described by a long-tailed ‘power law’.

Published in: Data & Analytics
  • Be the first to comment

Making Sense of Statistics in HCI: part 1 - wild and wide

  1. 1. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Making Sense of Statistics in HCI Part 1 – Wild and Wide concerning randomness and distributions Alan Dix http://alandix.com/statistics/
  2. 2. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix unexpected wildness of random just how random is the world? raindrops and horse races
  3. 3. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix a story In the far off land of Gheisra there lies the plain of Nali. For one hundred miles in each direction it spreads, featureless and flat, no vegetation, no habitation; except, at its very centre, a pavement of 25 tiles of stone, each perfectly level with the others and with the surrounding land. The origins of this pavement are unknown – whether it was set there by some ancient race for its own purposes, or whether it was there from the beginning of the world. Rain falls but rarely on that barren plain, but when clouds are seen gathering over the plain of Nali, the monks of Gheisra journey on pilgrimage to this shrine of the ancients, to watch for the patterns of the raindrops on the tiles. Oftentimes the rain falls by chance, but sometimes the raindrops form patterns, giving omens of events afar off. Some of the patterns recorded by the monks are shown on the following pages. Which are mere chance and which foretell great omens?
  4. 4. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix day 1
  5. 5. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix day 2
  6. 6. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix day 3
  7. 7. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix your choice? day 1 day 2 day 3 which are by chance ... and which are unusual?
  8. 8. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix why? ………………………………………. ………………………………………. ……………………………………….
  9. 9. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix really random empty squares & overfull squares
  10. 10. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix random but not uniform clumped towards the middle
  11. 11. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix too uniform every square has 5 rain drops too good to be true!
  12. 12. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix two horse races toss 20 coins add the heads to one row the tails to a second the winner is the first row to 10 before you start: what do you think will happen?
  13. 13. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix the race did you get a clear winner? or was it neck and neck?
  14. 14. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix the world is very random probability head = 0.5 number of heads ≠ number of tosses 2
  15. 15. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix lessons apparent differences may be chance real data has some bad values e.g. Mendel's sweet peas and electron discovery were too good!
  16. 16. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  17. 17. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix quick (and dirty!) tip for survey or other count data do square root times two
  18. 18. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Quick (and dirty!) Tip [1] estimate variation of survey data (for categories with small response) • survey 1000 people on favourite colour • say 10% of sample say it is blue • this represents 100 people • variation is roughly 2 x square root of 100 • that is about +/- 20 people, or +/- 2%
  19. 19. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Quick (and dirty!) Tip [2] For response nearer top end the same, but use the number who did not answer • Say 85% of sample of 200 say colour is green • 15% = 30 people did not say green • variation is roughly 2 x square root of 30 • that is about +/- 11 people, or +/- 5.5% • real value anything from 80 to 90 %
  20. 20. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Quick (and dirty!) Tip [3] If nearer 50:50 (e.g. presidential election!) then this is a little over estimate, use 1.5 instead • Say 50% of sample of 2000 say yes to question • 50% = 1000 people did not say green • variation is roughly 1.5 x square root of 1000 • that is about +/- 50 people, or +/- 2.5% N.B exact formula variance ~ n p (1-p)
  21. 21. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix … but really far more important: the fairness of the sample self-selection for surveys the phrasing of the question
  22. 22. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  23. 23. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix bias and non-independence is it fair? is it reliable?
  24. 24. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix bias systematic effects that skew results one way or other e.g. choice of LinkedIn vs. Snapchat survey WEIRD (Western, Educated, Industrialized, Rich, and Democratic) bias persists no matter your sample size but can sometimes be corrected e.g. sample variance … hence the n-1
  25. 25. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix bias vs. variability high variability with no bias – measurements equally likely to be more or less than real value, but may be far away in either direction – poor estimate of right thing low variability with bias – Measurements consistent with one another, but systematically one way – good estimate of wrong thing increase sample size or reduce variability eliminate bias or model it
  26. 26. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix independence different kinds of independence: • measurements (normal kind) • factor effects • sample prevalence non-independence often increases variability … but may cause bias too
  27. 27. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix independence – measurements measurements may be related to each other: order effects sequential measures on same person (learning, etc.) ‘day effects’ e.g. sunny day improves productivity experimenter effects your mood effects participants all increase variability
  28. 28. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix independence – factor effects Relationships and correlations between things you are measuring (or failing to measure) can confuse causality e.g. death rates often higher in specialist hospitals … because more sick people sent there even reverse effects … Simpson Paradox
  29. 29. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Simpson’s Paradox You run a course with some full-time and some part-time students You calculate: – average FT student marks go up each year – average PT student marks increase each year University complains your marks are going down Who is right?
  30. 30. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix Simpson’s paradox – you both are! FT students getting higher marks number of PT students increased
  31. 31. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix independence – sample the way you obtain your sample internal – subjects related to each other e.g. snowball sampling if not external – increases variance external – subject choice related to topic e.g. measuring age based on mobile app survey may create bias
  32. 32. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix crucial question Is the way I’ve organised my sample likely to be independent of the thing I want to measure? e.g. Fitts’ law with different colour targets. Are student subjects likely to behave differently than the general population?
  33. 33. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  34. 34. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix play! experiment with bias and independence http://www.meandeviation.com/tutorials/stats/morecoin.html
  35. 35. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix nos.of coins per row nos.of rows of coins press coin to start modify independence +ve more likely to be same -ve more likely to alternate set bias 0.5 = fair coin summary statistics area coins appear here row counts appear here
  36. 36. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix fair coin (independent tosses) but individual rows vary a lot average close to 50:50
  37. 37. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix biased coin (independent) average lower
  38. 38. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix positive correlation (runs) average 50:50 but variation higher note long runs see them swop!
  39. 39. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix negative correlation (alternation) average still 50:50 but variation lower largely alternate H/T very consistent
  40. 40. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix
  41. 41. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix distributions normal or not
  42. 42. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix stats 101 – what kind of data continuous: e.g. time to complete task (12.73 secs) discrete ... arithmetic: e.g. number of errors (average makes some sense) ordered/ordinal: .g. satisfaction rating (?average rating?) nominal/categorical: e.g. menu item chosen ( (File+Font)/2 = Flml ?)
  43. 43. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix bounded, tails, … the data number of heads in 6 tosses – discrete, finite number of heads until first tail – discrete, unbounded wait before next bus – continuous bounded below (zero), … but not above! difference between heights – continuous, unbounded your question is error rate higher – discrete, one tailed are completion times different – continuous, two tailed
  44. 44. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix approximations may approximate one type of distribution with another esp. using Normal https://commons.wikimedia.org/wiki/File:Binomial_Distribution.svg continuous / discrete bounded / unbounded
  45. 45. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix why is Normal normal? central limit theorem if you: – average lots of things (or near linearly combine) – around the same size (so none dominates) – nearly independent – and have finite variance then you get Normal distribition what?
  46. 46. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix non-Normal – what can go wrong? non-linearity – e.g. thresholds +ve / -ve feedback snowflakes and clouds bi-modal exam marks unbounded variance the more you sample the bigger the variation used to be rare e.g. wage/wealth distributions … but now Power Law …
  47. 47. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix earthquakes, sand piles, … and networks e.g. Facebook connections power law data is NOT Normal even when averaged power law – scale free https://en.wikipedia.org/wiki/Power_law long tail unbounded variance small number of frequent elements
  48. 48. Making Sense of Statistics in HCI: From P to Bayes and Beyond – Alan Dix

×