5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
Sturgis lund 2017
1. Why are we surprised when the polls are
wrong?
Professor Patrick Sturgis, University of
Southampton
Swedish Statistical Society Annual Conference
23 March 2017, Lund
2. What I’ll talk about
• Why we shouldn’t be surprised:
– Election polling methodology is difficult & prone to errors
– Election polling misses are more common than we think
– Election polls do not calculate or communicate sampling variability well
• A new procedure for calculating sampling variability
• It’s not just sampling variability though (Mean Squared Error)
2
4. The final UK polls 2015
Pollster Mode Fieldwork n Con Lab Lib UKIP Green Other
Populus O 5–6 May 3,917 34 34 9 13 5 6
Ipsos-MORI P 5–6 May 1,186 36 35 8 11 5 5
YouGov O 4–6 May 10,307 34 34 10 12 4 6
ComRes P 5–6 May 1,007 35 34 9 12 4 6
Survation O 4–6 May 4,088 31 31 10 16 5 7
ICM P 3–6 May 2,023 34 35 9 11 4 7
Panelbase O 1–6 May 3,019 31 33 8 16 5 7
Opinium O 4–5 May 2,960 35 34 8 12 6 5
TNS UK O 30/4–4/5 1,185 33 32 8 14 6 6
Ashcroft* P 5–6 May 3,028 33 33 10 11 6 8
BMG* O 3–5 May 1,009 34 34 10 12 4 6
SurveyMonkey* O 30/4-6/5 18,131 34 28 7 13 8 9
Result 37.8 31.2 8.1 12.9 3.8 6.3
MAE (=1.9) 4.1 2.5 1.0 1.4 1.4 0.9
4
7. Methodology of UK election
polls
i. Collect quota sample of individuals, with quota targets for
marginal distributions of some variables 𝐗∗
(e.g. age, sex,
region)
ii. Derive post-stratification weights 𝑤𝑖
∗
for respondents i, so
weighted marginal distributions of 𝐗 match population targets
iii. Assign each respondent a probability 𝑃 𝑇𝑖 that respondent will
vote in the election (𝑇𝑖=1) in the election
i. Usually based on L, self-assessed likelihood to vote
iv. Estimate shares of vote in the election (𝑃𝑖) as weighted
proportions of self-reported vote intention (𝑉𝑖) with weights
𝑤𝑖= 𝑤𝑖
∗
𝑃 𝑇𝑖
7
8. Assumptions required for unbiased estimates
1. Representative sampling: p(V, L|X) is the same in the sample as
in the population
2. Correct model for turnout probabilities: Assigned turnout weights
𝑃 𝑇𝑖 are equal to p(𝑇𝑖=1|Vi,Li, Xi) in the population
3. Stated vote intention is equal to actual vote:
p(V |T=1) = p(P|T=1)
8
9. Where to get turnout probabilities from?
• Main approach is to ask respondents:
– On a scale of 1-10 how likely is it that you will vote?
– Sometimes supplemented with other questions e.g. on whether voted in
last election, importance of voting, and so on
• Convert responses to turnout weights
• Alternatively, use historical data containing a measure of turnout
• Build prediction model using respondent characteristics
• Use model parameters to produce predictions of future turnout on
new sample
9
14. Election polls do not calculate or communicate
sampling variability well
14
15. ‘margin of error’
• Many polls only report point estimates of vote shares
• Some report ‘margin of error’, usually +/- 3% for each party share
• This is based on 95% confidence interval for a proportion under
simple random sample (n=1000)
• Invariant to sample size (voters are full sample?), point estimates,
quotas, and post-stratification
15
16. Consequences
• Reporting of small differences in vote shares as though significant,
‘Party X surges to a 1 point lead’
• Which, in turn, creates a (false) belief amongst public &
commentators that polls have a high degree of acuity
• Dislocation between research design and research objectives
(arbitrary sample sizes given expected effect size)
16
17. 2015 Polling Inquiry Recommendations 11 & 12
• Pollsters should:
– provide confidence (or credible) intervals for each separately listed party in
their headline share of the vote.
– provide statistical significance tests for changes in vote shares for all listed
parties compared to their last published poll.
• “clearer communication of the uncertainty around poll estimates that
better reflects the underlying research design, as well as greater
transparency in how the estimates of uncertainty are produced” Sturgis
et al 2016, p77
17
18. A bootstrap approach
• Assumption: sample is random draw from distribution of repeated
samples of same design:
1. Draw bootstrap resamples (with replacement) in a way which mimics the
quota sampling
2. Estimate vote shares for each sample using same estimation procedure
as for real sample
3. Use variation across estimates to estimate sampling variation
18
19. 19
Sampling intervals for Con-Lab
difference (%) UK 2015 General election
Pollster Con-Lab(%)
(electionresult=+6.5%)
Estimate 95% interval* N
Populus -0.1 (-2.5; +2.0) 3695
Ipsos-MORI -0.3 (-6.6; +6.1) 928
YouGov +0.4 (-1.1; +1.8) 9064
ComRes +0.8 (-4.6; +6.3) 852
Survation +0.1 (-2.2; +2.5) 3412
ICM +0.0 (-2.8; +3.1) 1681
Panelbase -2.7 (-5.6; +0.2) 3019
Opinium +0.4 (-1.8; +2.5) 2498
TNS UK +0.8 (-3.6; +5.2) 889
* Adjustedpercentile intervals,calculatedusing
10,000 bootstrapsamples.
No estimate contains true value
in 95% interval
20. 20
Design effects under different scenarios
Similar pattern for design variances as
in probability sampling, with quotas
in the role of strata & post-stratification
weights as weights
Red=past vote as quota; green=past vote as weight; blue= no past vote
21. Mean Square Error
• This procedure measures only error due to sampling variability
• Some would argue the more important errors are sample bias
(violations of assumption 1 and 2)
• Incorporate bias and variance for measure of mean squared
error?
• Not obvious how one would do this
• Best left to poll aggregators & modelers
21
22. Concluding remarks
• Polls are judged by whether they predict election outcome not by
statistical measures of error
• Polls can be right for the wrong reasons (self-cancelling errors)
• Pollsters should do better job of communicating sampling error
• But we will probably still be surprised by the next polling miss!
22
