Successfully reported this slideshow.
Upcoming SlideShare
×

# Polling data to assess election campaigns: Why polling aggregation helps you avoid cherrypicking polls and falling for outliers

34 views

Published on

Presentation by Cornelius Hirsch at the 2019 CMPF Summer School for Journalists and Media Practitioners - Covering Political Campaigns in the Age of Data, Algorithms & Artificial Intelligence

Published in: Education
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Polling data to assess election campaigns: Why polling aggregation helps you avoid cherrypicking polls and falling for outliers

1. 1. 1
2. 2. OUTLINE: 1. Introduction 2. Polling errors 3. How to identify good and bad polls 4. Important concepts around polls 5. Poll of polls 6. The 2019 European Parliament elections on politico.eu 7. How to report polls
3. 3. POLLS HAVE A BAD REPUTATION WITH GOOD REASON? 3 INTRODUCTION
4. 4. THE POLLS ARE AL(L)RIGHT 02-04 4
5. 5. THE POLLING ERROR 1. Can be expected to be around 2-2.2 percentage points 2. Is very consistent over time and across elections. 3. Can be factored in. 4. Has to be clearly communicated. 5 THE POLLS ARE AL(L)RIGHT Source: Jennings and Wlezien (2018) https://doi.org/10.1038/s41562-018-0315-6
6. 6. POLL ERRORS ACROSS ELECTORAL HISTORY a: the absolute error for all parties/candidates. • Hollow grey circle: error for each party • Black circle: mean absolute error across all parties b: the mean absolute error. c: the mean absolute error on the margin. d: the mean absolute value of the log odds ratio. e: the variance of the mean absolute error. f: the variance of the mean absolute error 6 THE POLLS ARE AL(L)RIGHT Source: Jennings and Wlezien (2018) https://doi.org/10.1038/s41562-018-0315-6
7. 7. POLLING ERROR FOR THE AUSTRIAN PARLIAMENTARY ELECTION • Average 1.35 percentage points • Polls with larger sample size and shorter field period perform better 7 THE POLLS ARE AL(L)RIGHT
8. 8. POLLING ERROR IN DENMARK • 1.36 percentage points polling error • Timing matters • Method matters • Quality matters 8 THE POLLS ARE AL(L)RIGHT
9. 9. 9 BUT WHAT ABOUT TRUMP AND BREXIT? Brexit: • The day before the referendum poll of polls showed 48% for remain 45% for leave, but 10 undecided! THE POLLS ARE AL(L)RIGHT Source: https://en.wikipedia.org/wiki/Opinion_polling_for_the_United_Kingdom_European_Union_membership_referendum A CLOSER LOOK 2016 US Presidential election: • Chances 70:30 – trump would win every third election. Would you get on a plane if every third would crash? • National polls were correct for popular vote, but electoral college is what matters. • Polls in key swing states with above average polling error overall 5 percentage points absolute error on average. • Trump won the election with very small margin in important states.
10. 10. 10 BUT WHAT ABOUT TRUMP AND BREXIT? Brexit: • The weeks before the referendum poll of polls showed 48% for remain 45% for leave, but 10 undecided! All polls: THE POLLS ARE AL(L)RIGHT Source: https://en.wikipedia.org/wiki/Opinion_polling_for_the_United_Kingdom_European_Union_membership_referendum A CLOSER LOOK
11. 11. 11 HOW TO IDENTIFY GOOD AND BAD POLLS 6 KEY CRITERIA • Source: In most cases a news outlet commissions the polls and publishes it in an article, trustworthy newspaper? • Field work: When was the poll conducted? • Sample size: How many people where interviewed? Rule of thumb: >800 • Methodology: Was the poll conducted online, in face-to-face interviews or via telephone? • Number of undecided voters: Do the figures add up to 100% or is the % of undecided published? THE POLLS ARE AL(L)RIGHT
12. 12. “REPRESENTATIVE” 12 • Each person in the group in question has the same change to be interviewed. • Has nothing to do with sample size • Has nothing to do with population size either • How to chose representative sample? • Two methods: random sampling or quota sampling • E.g. randomly drawn phone numbers • Set quotas for gender, age, etc., which have to be met in the polling region, used in online panels. • Weighting for underrepresented , non-response adjustment. A good (=representative) small sample is better than a bad large sample! To taste a soup you don’t have to empty the pot, a spoonful will do! IMPORTANT CONCEPTS WHEN REPORTING ABOUT POLLS THE POLLS ARE AL(L)RIGHT
13. 13. THE MARGIN OF ERROR CONFIDENCE INTERVAL 13 • tells you how many percentage points your results will differ from the real population value. • the range of values below and above the sample statistic in a confidence interval. • For simple random sample approximated: • For the percentage value “p” of a specific party in a poll • Therefore, smaller for smaller parties. • a way to show what the uncertainty is with a certain poll. • E.g. 1000 people interviewed, party A at 33% → Margin of error is +/- 2.9 percentage points → Confidence interval of 30.1 and 35.9. • if the poll is repeated using the same techniques, 95% of the calculated intervals will include the “true” population value. IMPORTANT CONCEPTS WHEN REPORTING ABOUT POLLS THE POLLS ARE AL(L)RIGHT Note: https://www.langerresearch.com/moe/ offers a margin of error calculator 𝑀𝑜𝐸 ≅ 0.98 √𝑛 𝑀𝑜𝐸 = 1.96 0.33(1 − 0.33) 1000 = 2.91% → +/- MoE → Confidence Interval 𝑀𝑜𝐸 = 1.96 𝑝(1 − 𝑝) 𝑛
14. 14. 14 OTHER ERRORS THE POLLS ARE AL(L)RIGHT Other sources of potential bias: • Sampling Error – because you don’t poll the entire population • Coverage Error – some groups can’t be polled • Measurement Error – bad wording of question, bad instruments etc. • Non-Response Error – bias because of who responds to poll and who doesn’t. • All together = Total Survey Error
15. 15. 15 AN EXAMPLE FOR THE MEASUREMENT ERROR THE POLLS ARE AL(L)RIGHT Source: https://twitter.com/alemannoEU
16. 16. WHY A POLL OF POLLS IS SUPERIOR 05 16
17. 17. PARABLE OF THE THREE RINGS (OR POLLSTERS) Which poll is the one to trust? 17 WHICH POLL IS THE ONE TO TRUST? #POLLOFPOLLS
18. 18. IS THERE A HOUSE EFFECT? • Polls ahead of the Swedish parliament election 2018 by pollster reveal significant differences. • YouGov and Sentio constantly overestimated Sweden democrats. • Their method? – Online polls only • In 2014 all pollsters underestimated Sweden Democrats, YouGov and Sentio to a lesser extent • Shy Sweden Democrats-supporters? • Less reluctant to reveal voting intention in “anonymous” online polls? • In 2018 overcorrected or did response behavior change? 18 #POLLOFPOLLS
19. 19. COMBINING POLLS REVEALS TRENDS AND OUTLIERS 19 #POLLOFPOLLS Source: pollofpolls.eu/FR NOT ONLY FOR VOTING INTENTION • Don’t cherry-pick one poll, where there are more. • Our aggregation model uses the so-called Kalman filtering approach + smoothing to track the trends in the polls. Kalman filter can be explained with “GPS”-analogy: • Support for a party is the “true” location • In your Map-app, the blue circle around your GPS-location is the uncertainty, every new signal (poll) from the satellite (polling firm) improves the location and who will find your way.
20. 20. THE 2019 EUROPEAN PARLIAMENT ELECTIONS 06 20
21. 21. POLITICO EUROPEAN ELECTIONS HUB 21 #EP2019 Source: politico.eu/EU2019
22. 22. POLITICO EUROPEAN ELECTIONS HUB • The only daily updated seat estimate • Interactive • Coalition builder • Figures with and without UK • Filter by country • Map view 22 #EP2019 Source: politico.eu/EU2019
23. 23. OUR SEAT PROJECTION OVER TIME 23 #EP2019 • Relatively stable • Clear from beginning that EPP+S&D < 50% of seats • Biggest shifts due to group changes of individual parties • Brexit analysis • Challenge was how to deal with new groups and (unofficially) announced group changes.
24. 24. EUROPEAN PARLIAMENT ELECTION RESULTS 24 #EP2019 Source: politico.eu/EU2019 28 COUNTRY PAGES
25. 25. 25 HOW ACCURATE WERE THE POLLS? • Overall very good • Big upsets in Slovakia (PS+SPOLU), Lithuania (LVZS) • Green parties underestimated in Ireland, Austria, Germany. • Almost no polls in the Netherlands • No polls in Luxembourg • No polls after Ibizagate broke in Austria #EP2019 POLLING ERROR ACROSS ALL 28 EU COUNTRIES
26. 26. EUROPEAN PARLIAMENT ELECTION RESULTS 26 #EP2019 Source: politico.eu/EU2019 LAST UPDATED 21 JUNE
27. 27. HOW TO REPORT POLLS 07 27
28. 28. HOW TO REPORT POLLS 1. Check for key criteria 2. Report key criteria (polling firm, commissioned by, field period, sample size, method) 3. Look at figures with margin of error in mind 4. Report trend compared to last poll from same polling firm 5. Don’t overinterpret small number 1. One percentage point lead is not a lead 2. One percentage point change is not a real change but statistical noise 28 #CASESTUDY
29. 29. 29 FROM POLLS TO HEADLINES “Italians in 2019 hate chicken, Greek salad overtaking fish!” When looking at polling results, always look for the PQRST: P. Polling Firm: Cornelius Hirsch? Never heard of that polling firm, also can’t find a website or another published poll. Q. Question: Chicken was not polled, international respondents. R. Results: Others and/or undecided are not reported S. Sample size: <800 people polled, <400 declared voting intention T. Timing: One day polling period in June 2019. #CASESTUDY
30. 30. 30 FROM POLLS TO HEADLINES (Bloomberg) — Greece's main opposition new democracy party would win parliamentary elections with a 9 percentage point lead if they were held now, according to a pulse RC poll for Skai TV. New democracy, a conservative party, would get from 31.5% to 36.5% of the vote compared with 22.5% to 27.5% for the governing leftist SYRIZA party, according to the poll. The survey is the first since prime minister Alexis Tsipras called a snap national ballot following his party’s defeat in European elections a week ago. The center-left movement for change would come third with 6.5% to 9.5% of the vote, the communist party of Greece fourth with 4.5% to 6.6%, the far-right golden dawn party fifth with 4% to 6% and Greek solution, a new nationalist party, placed sixth with 3% to 5%. Mera25, the new party of former finance minister Yanis Varoufakis would receive from 2% to 4% of the vote. Parties need to reach the threshold of 3% to elect a member to parliament. Pulse RC surveyed 1,358 people from may 30 to June 1. The margin of error was +/- [XY] Calculate margin of error! Key characteristcs? #CASESTUDY Source: https://nationalpost.com/news/world/new-democracy-wins-first-poll-since-tsiprass-snap-election-call INVENT A TITLE
31. 31. POLL OF POLLS FOR THE 2019 GREEK SNAP ELECTION 31 #CASESTUDY Source: pollofpolls.eu/GR ONE POLL CAN BE AN OUTLIER, IT’S ALWAYS BETTER TO LOOK AT THE POLL OF POLLS Interpretation? Significant lead? → Larger than 2 x MoE (conservative estimate 2 percentage points) → Yes Increasing trend? In Jan – May 2019 ND around 36-37%, now 39% → “Slightly” Golden Dawn significantly decreasing. Electoral threshold in Greece of 4%, small parties significantly below or above?
32. 32. 32 RECAP Polls are …accurate, with an average margin of error of around 2 percentage points. …only a snapshot in time. …more insightful when aggregated, revealing trends and outliers. …one tool of many to assess an election campaign. …like a fine perfume, you should smell it, but not drink it. Surveys on twitter or on websites, where anyone can respond as often as they like, are “clickers” – they are not “polls”. To Remember • Margin of Error depends on sample size and not on population size. • Confidence Interval shows uncertainty. • Representativeness depends on chance to get interviewed, not on sample size or population size.
33. 33. THANK YOU FOR YOUR ATTENTION LET’S STAY IN TOUCH @VollCornHirsch chirsch@politico.eu pollofpolls@politico.eu
34. 34. POLLING ERROR BY PARTY 34 #EP2019