Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Sturgis lund 2017

60 views

Published on

Sturgis

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Sturgis lund 2017

  1. 1. Why are we surprised when the polls are wrong? Professor Patrick Sturgis, University of Southampton Swedish Statistical Society Annual Conference 23 March 2017, Lund
  2. 2. What I’ll talk about • Why we shouldn’t be surprised: – Election polling methodology is difficult & prone to errors – Election polling misses are more common than we think – Election polls do not calculate or communicate sampling variability well • A new procedure for calculating sampling variability • It’s not just sampling variability though (Mean Squared Error) 2
  3. 3. 3 I’ll eat my hat/kilt if the exit poll is right
  4. 4. The final UK polls 2015 Pollster Mode Fieldwork n Con Lab Lib UKIP Green Other Populus O 5–6 May 3,917 34 34 9 13 5 6 Ipsos-MORI P 5–6 May 1,186 36 35 8 11 5 5 YouGov O 4–6 May 10,307 34 34 10 12 4 6 ComRes P 5–6 May 1,007 35 34 9 12 4 6 Survation O 4–6 May 4,088 31 31 10 16 5 7 ICM P 3–6 May 2,023 34 35 9 11 4 7 Panelbase O 1–6 May 3,019 31 33 8 16 5 7 Opinium O 4–5 May 2,960 35 34 8 12 6 5 TNS UK O 30/4–4/5 1,185 33 32 8 14 6 6 Ashcroft* P 5–6 May 3,028 33 33 10 11 6 8 BMG* O 3–5 May 1,009 34 34 10 12 4 6 SurveyMonkey* O 30/4-6/5 18,131 34 28 7 13 8 9 Result 37.8 31.2 8.1 12.9 3.8 6.3 MAE (=1.9) 4.1 2.5 1.0 1.4 1.4 0.9 4
  5. 5. 5
  6. 6. Election polling is difficult… 6
  7. 7. Methodology of UK election polls i. Collect quota sample of individuals, with quota targets for marginal distributions of some variables 𝐗∗ (e.g. age, sex, region) ii. Derive post-stratification weights 𝑤𝑖 ∗ for respondents i, so weighted marginal distributions of 𝐗 match population targets iii. Assign each respondent a probability 𝑃 𝑇𝑖 that respondent will vote in the election (𝑇𝑖=1) in the election i. Usually based on L, self-assessed likelihood to vote iv. Estimate shares of vote in the election (𝑃𝑖) as weighted proportions of self-reported vote intention (𝑉𝑖) with weights 𝑤𝑖= 𝑤𝑖 ∗ 𝑃 𝑇𝑖 7
  8. 8. Assumptions required for unbiased estimates 1. Representative sampling: p(V, L|X) is the same in the sample as in the population 2. Correct model for turnout probabilities: Assigned turnout weights 𝑃 𝑇𝑖 are equal to p(𝑇𝑖=1|Vi,Li, Xi) in the population 3. Stated vote intention is equal to actual vote: p(V |T=1) = p(P|T=1) 8
  9. 9. Where to get turnout probabilities from? • Main approach is to ask respondents: – On a scale of 1-10 how likely is it that you will vote? – Sometimes supplemented with other questions e.g. on whether voted in last election, importance of voting, and so on • Convert responses to turnout weights • Alternatively, use historical data containing a measure of turnout • Build prediction model using respondent characteristics • Use model parameters to produce predictions of future turnout on new sample 9
  10. 10. Election poll misses are more common than we think 10
  11. 11. Net error, final polls - Conservative vote share 1945- 2015 11
  12. 12. Net error, final polls - Labour vote share 1945-2015 12
  13. 13. Net error, final polls - Conservative lead 1945-2015 13
  14. 14. Election polls do not calculate or communicate sampling variability well 14
  15. 15. ‘margin of error’ • Many polls only report point estimates of vote shares • Some report ‘margin of error’, usually +/- 3% for each party share • This is based on 95% confidence interval for a proportion under simple random sample (n=1000) • Invariant to sample size (voters are full sample?), point estimates, quotas, and post-stratification 15
  16. 16. Consequences • Reporting of small differences in vote shares as though significant, ‘Party X surges to a 1 point lead’ • Which, in turn, creates a (false) belief amongst public & commentators that polls have a high degree of acuity • Dislocation between research design and research objectives (arbitrary sample sizes given expected effect size) 16
  17. 17. 2015 Polling Inquiry Recommendations 11 & 12 • Pollsters should: – provide confidence (or credible) intervals for each separately listed party in their headline share of the vote. – provide statistical significance tests for changes in vote shares for all listed parties compared to their last published poll. • “clearer communication of the uncertainty around poll estimates that better reflects the underlying research design, as well as greater transparency in how the estimates of uncertainty are produced” Sturgis et al 2016, p77 17
  18. 18. A bootstrap approach • Assumption: sample is random draw from distribution of repeated samples of same design: 1. Draw bootstrap resamples (with replacement) in a way which mimics the quota sampling 2. Estimate vote shares for each sample using same estimation procedure as for real sample 3. Use variation across estimates to estimate sampling variation 18
  19. 19. 19 Sampling intervals for Con-Lab difference (%) UK 2015 General election Pollster Con-Lab(%) (electionresult=+6.5%) Estimate 95% interval* N Populus -0.1 (-2.5; +2.0) 3695 Ipsos-MORI -0.3 (-6.6; +6.1) 928 YouGov +0.4 (-1.1; +1.8) 9064 ComRes +0.8 (-4.6; +6.3) 852 Survation +0.1 (-2.2; +2.5) 3412 ICM +0.0 (-2.8; +3.1) 1681 Panelbase -2.7 (-5.6; +0.2) 3019 Opinium +0.4 (-1.8; +2.5) 2498 TNS UK +0.8 (-3.6; +5.2) 889 * Adjustedpercentile intervals,calculatedusing 10,000 bootstrapsamples. No estimate contains true value in 95% interval
  20. 20. 20 Design effects under different scenarios Similar pattern for design variances as in probability sampling, with quotas in the role of strata & post-stratification weights as weights Red=past vote as quota; green=past vote as weight; blue= no past vote
  21. 21. Mean Square Error • This procedure measures only error due to sampling variability • Some would argue the more important errors are sample bias (violations of assumption 1 and 2) • Incorporate bias and variance for measure of mean squared error? • Not obvious how one would do this • Best left to poll aggregators & modelers 21
  22. 22. Concluding remarks • Polls are judged by whether they predict election outcome not by statistical measures of error • Polls can be right for the wrong reasons (self-cancelling errors) • Pollsters should do better job of communicating sampling error • But we will probably still be surprised by the next polling miss! 22
  23. 23. Thank you Working paper available: http://eprints.ncrm.ac.uk/4009/ 23

×