Great Variation Among Pollsters 2 The Death of Polling?

The Death of Polling Version 1 Public 1 Death of Polling? The #ipsosmorilive

The Death of Polling Version 1 Public 2 Ben Page, Ipsos MORI Agenda Julia Clark, Ipsos USA David Ahlin, Ipsos Sweden Nando Pagnoncelli, Ipsos Italy Darrell Bricker, Ipsos Canada Q&A

The Death of Polling Version 1 Public 3 Ben Page Ipsos MORI ben.page@ipsos.com #ipsosmorilive

4The Death of Polling? Version 1 Public Sadly not everywhere looks like this … Average candidate error in US Presidential Elections (INDIV CAND SHARE) 1936 6% 1.5% 2012

5The Death of Polling? Version 1 Public There’s a global Ipsos has carried out election conversation going on about polling polling in c30 countries since 2007

6The Death of Polling? Version 1 Public And pollsters all around the world are How do we achieve a representative sample? How do we predict turnout accurately? having to face up to hard questions How do we make best use of the increasing range of methodologies open to us? How do we ensure our polls are reported well and understood among the media, politicians and the public?How do we avoid other biases, such as social desirability?

7The Death of Polling? Version 1 Public ... although our answers Size/diversity of population Turnout levels/compulsory voting Stable party system or new insurgents? Generational declines in party loyalty Societal/cultural context Media expectations vs budgets and levels of polling transparency? Survey researchers or moving towards big data modelling? National polls vs local/state-level polling? Traditional face-to-face methods vs new or mixed modes? And in any one country likely to be good and bad examples of each! may be very different!

The Death of Polling Version 1 Public 8 UK experience The

9The Death of Polling? Version 1 Public British Polling Council Selection of samples (their main explanation) Representativeness of sample (key among explanations) Correction for likelihood of voting (less important) Late swing (some signs, but not that important) Herding (unproven) Inquiry Interim findings

10The Death of Polling? Version 1 Public Our final poll – all parties less than 2% points away How would you vote if there were a General Election tomorrow? from actual – except Labour, overestimated 36% 35% 11% 5% 8% 5% Ipsos MORI final poll GB final result Conservative lead = +1 Conservative lead = +6.5 CONSERVATIVE LABOUR UKIP GREEN LIB DEM OTHER 37.7% 31.2% 12.9% 3.8% 8.1% 6.4%

11The Death of Polling? Version 1 Public ‘Shy Tories’ not our problem – instead too How would you vote if there were a General Election tomorrow? many Labour voters and not enough non-voters Conservatives Labour Non voters 11.3m 12.5m 9.3m 12.2m 20.5m 15.4m Actual Implied from final poll

The Death of Polling Version 1 Public 12 Ipsos MORI’s view: We need to take a two-pronged approach – tackle the problem of more politically engaged taking part and also making sure we can detect differential shift in turnout over-claim between parties

The Death of Polling Version 1 Public 13 Ipsos MORI’s view: Healthy scepticism not the death of polling

14The Death of Polling? Version 1 Public We need to improve representation of No easy answers to this (especially bearing in mind budget and time Constraints) politically disengaged/non-voters in our samples Already introduced newspaper weighting (for example to reduce proportion of broadsheet readers). In four months from September to December 2015 this: • Reduced the proportion of claimed likely voters by an average of 3 percentage points a month • Primarily at the expense of the Labour share (down on average by 3 points, Conservatives up by 1.75 points) But will continue other experiments (for example changes in quotas, and so on)

15The Death of Polling? Version 1 Public Random samples after election though do not appear to be hugely more representative than our quota ones Election result Final telephone polls (average) Final Ipsos MORI poll British Social Attitudes British Election Study1 Ipsos MORI post- GE past vote (June-July) Voting intentions2 Difference from result Voting intentions Difference from result Report of vote Difference from result Report of vote Difference from result Report of vote Difference from result % % % % % % Con. 37.7 34.5 -3.2 36 -1.7 39.7 +2.0 40.6 +2.9 37.9 +0.2 Labour 31.2 34.3 +3.1 35 +3.8 33.6 +2.4 32.7 +1.5 32.5 +1.3 Other parties 31.2 31.2 0.0 29 -2.2 26.7 -4.5 26.7 -4.5 29.6 -1.6

The Death of Polling Version 1 Public 16 The Death of Polling? So much more than a horse race

17The Death of Polling? Version 1 Public So much more richness in the polls to help us understand public opinion

18The Death of Polling? Version 1 Public Not to mention giving voters a chance to express their views

19The Death of Polling? Version 1 Public Or using twitter analytics to get live, real-time reactions to the big events: 239,000 tweets in the 2nd debate Twitter ‘worm’ – real time analysis of reaction to second leader debate (2,500+ per minute)

20The Death of Polling? Version 1 Public And using new digital techniques to get closer to voters 7426 posts across 340 forum topics c.2000 members from across the UK Over 7500 survey responses

The Death of Polling Version 1 Public 21 Julia Clark Ipsos USA julia.clark@ipsos.com #ipsosmorilive

THE THEME IS:

More Data Than Ever 0 20 40 60 80 100 120 Jan-60 Dec-60 Nov-61 Oct-62 Sep-63 Aug-64 Jul-65 Jun-66 May-67 Apr-68 Mar-69 Feb-70 Jan-71 Dec-71 Nov-72 Oct-73 Sep-74 Aug-75 Jul-76 Jun-77 May-78 Apr-79 Mar-80 Feb-81 Jan-82 Dec-82 Nov-83 Oct-84 Sep-85 Aug-86 Jul-87 Jun-88 May-89 Apr-90 Mar-91 Feb-92 Jan-93 Dec-93 Nov-94 Oct-95 Sep-96 Aug-97 Jul-98 Jun-99 May-00 Apr-01 Mar-02 Feb-03 Jan-04 Dec-04 Nov-05 Oct-06 Sep-07 Aug-08 Jul-09 Jun-10 May-11 Apr-12 Number of Polls per month - 1960 - 2013Number of Polls per Month – 1960 - 2013 17,058 polls in 2012

Current US Political Polling Methodologies PHONE RDD Lists IVR (Robo) TRADITIONAL ONLINE Probability Panel Only Lists Blended NONTRADITIONAL ONLINE SurveyMonkey Google

Great Variation Among Pollsters 2 5

Constantly Changing Questionnaire (daily) Election Day ‘Exit Poll’ of 40,000 Voters Continual Survey: 11,000/month (24/7/365) Daily Assessing Events Same-Day (Parsing) 26 State-level polls (2k) with rolling reporting What Does Ipsos Do?

A VERY QUICK WORD ON THE US ELECTION…

ANTI-ESTABLISHMENT THEME: But Differential Framing of Problems & Causes The rich are to blame Middle Class Economics Americans First Restarting American Exceptionalism VS.

Trump & Sanders are the Response 17% 5% 5% 6% 13% 3% 3% 29% 19% Trump Cruz Carson Otheroutsiders(Paul, Fiorina,etc) Establishment(Bush, Rubio,etc) Wouldn'tvote O'Malley Clinton Sanders Republicans (46%) Democrats (51%) 46% of Americans are supporting “nontraditional” candidates

Political Fundamentals Speak to a Republican Year 65% 60% 55% 50% 45% 40% 35% Government Approval Rating OddsofWinning Incumbent Party’s Odds of Winning White House SUCCESSORS INCUMBENTS (2012)

… AND LOW TURNOUT BENEFITS THE REPUBLICANS TOO Source: Reuters / Ipsos Poll; Sept-Oct 2014 30% 35% 40% 45% 50% 55% 60% 65% 51 50 49 49 48 47 47 47 44 44 45 46 46 48 Republican Vote Democratic Vote 2012 TurnoutActual 2014 Turnout GenericCongressionalVoteShare Turnout Level (by Likely Voter Model) 2014 Generic Congressional Ballot by Turnout Levels

The Death of Polling Version 1 Public 33 Nando Pagnoncelli Ipsos Italy nando.pagnoncelli@ipsos.com #ipsosmorilive

35 © 2015 Ipsos. A GAME CHANGER EFFECT THE EVOLUTION OF THE ELECTOR ELECTORATES become fluid, reactive to the political offer, they lose their sense of belonging 2013 is the end of a 20-year period of almost perfect turnover between centre-left and centre-right, during which • neither coalition was in office for two turns in a row • voters switches between the two coalitions were residual 3

38 © 2015 Ipsos. EVERYBODY BECAME A POLLING EXPERT, except the experts THE POLLSTERS UNDER ACCUSATION Discussions went on for weeks on the limits of polls  Methodologies  Sample size  Coverage  Bias  … were discussed and commented and criticised by anybody and all on TV, in newspapers, online forum, social networks, workplaces and cafés 6

39 © 2015 Ipsos. Tecnè Demos Ipsos diff vs. actual results RIVOLUZIONE CIVILE 0,9% 0,9% 1,0% SEL 0,0% 0,4% 0,4% PD 4,9% 3,7% 4,9% ALTRI CENTRO SINISTRA -0,4% -0,5% -0,4% TOTALE CENTROSINISTRA 4,5% 3,6% 4,9% CON MONTI PER L'ITALIA -0,9% 2,0% 0,4% UDC 0,8% 1,0% 0,7% FLI 0,0% 0,2% 0,2% TOTALE CENTRO -0,1% 3,2% 1,3% LEGA NORD 1,0% 1,2% -0,1% PDL -0,3% -1,5% -1,4% ALTRI CENTRO DESTRA -0,7% -0,7% 0,6% TOTALE CENTRODESTRA 0,0% -1,0% -0,9% MOVIMENTO 5 STELLE BEPPEGRILLO.IT -5,8% -6,6% -5,6% POLLSTERS WERE INACCURATE THE POLLSTERS UNDER ACCUSATION WHAT WENT WRONG WHY IT WENT WRONG For all the major agencies in Italy • overestimated the Democratic Party’s • underestimated the M5S result • No past voters behaviour on M5S, which is a key component of the weighting process • The difficulties of intercepting the potential M5S voters • reticence in centre-left supporters to declare their intention • last minute swing (25% estimated to have decided 2 days before) • A high refusal rate, affecting differently the various groups of respondents (10 contacts to get a valid interview) • Respondents lie 7

40 © 2015 Ipsos. BUT SOME WERE MORE ACCURATE THAN OTHERS THE POLLSTERS UNDER ACCUSATION WHAT WENT RIGHT WHY IT WENT RIGHT IPSOS correctly described the scenario • The high level of abstention (26,5%) • The dramatic growth of M5S • The victory of the centre-left at the Chamber of Deputies • The hung Senate • All the other parties’ results Because • A mixed sampling method was used • Sample sizes were large enough • We had been polling continuously – weekly, even daily – for a long time • We had a wealth of data • We had lots of analyses on targets, geographical areas, voters switch dynamics 8

41 © 2015 Ipsos. THE PREDOMINANCE OF MADNESS THE POLLSTERS CHALLENGE Our stakeholders show a schizophrenic attitude • one minute, they attack us, because we couldn’t predict the future as exactly as we are expected to • the minute after, they ask for data, comments, explanations, because they need our expertise in interpretation We would like to see expectations decrease, towards the predictive nature of our work but we know it’s inconceivable 9

42 © 2015 Ipsos. THE FlashForward Effect THE POLLSTERS CHALLENGE … and everyone acted consequently. Polls act as an activator of two powerful, contrasting forces in electors • Self-fulfilling prophecies vs. • Self-defeating prophecies “On October 6, the planet blacked out for two minutes and seventeen seconds. The whole world saw the future...” 10

The Death of Polling Version 1 Public 43 David Ahlin Ipsos Sweden david.ahlin@ipsos.com #ipsosmorilive

2,9 (2006) 5,7 (2010) 12,9 (2014) 0 10 20 30 Electoral support for the Sweden Democrats 2006-2016 2006 2010 2014

0 30 60 Left-Right Parties Election 2006 Left-Right Parties February 2016 From two political blocks to three – in ten years

Immigration most pressing issue and Voter support for Sweden Democrats 5 8 13 20 20 40 6 7 9 12 14 17 0 20 40 June-2010 June-2014 Aug-2014 Jan-2015 Jun-2015 Jan-2016 Immigration/integration most pressing issue Voter support for the Sweden Democrats

Immigration most pressing issue and google searches for “refugee” in Swedish 8 13 20 20 40 0 20 40 60 80 100 June-2014 Aug-2014 Jan-2015 June 2015 Jan-2016 Number of people say immigration/integration issues are most pressing issue The number of searches on the word ”Refugee” on Google in Swedish during the same time period

17.2 23.9 0 20 Average voter support Telephone based methods Average voter support Web based methods Voter support Sweden Democrats based on method of measurement Feb 2016

49 © 2015 Ipsos. The state of polling 2016 SWEDEN Social effects / Interviewer effects Higher share of non- response in strong Sweden Democrat regions From 45 % – 25 % in the last six years SOCIAL EFFECTS AND NON-RESPONSE DROPPING RESPONSE RATES Less than 50 per cent with younger Swedes Not a problem with seniors SAMPLE COVERAGE Telephone calls only for close friends and family? (Pew 2015) Web based and telephone based give different answers Transparency! Mobile first strategy? NEW BEHAVIORS NEW METHODS

The Death of Polling Version 1 Public 50 Darrell Bricker Ipsos Canada darrell.bricker@ipsos.com #ipsosmorilive

51 © 2015 Ipsos. Ipsos Final Poll vs. Actual Election Results—National ELECTION DEBRIEF Ipsos Final Poll Election Results 38% 40% 31% 32% 22% 20% 4% 5% 4% 4% Base: Final Call Poll (Decided Voters, Leaners Included n=2,226; Online n=1,328; CATI n=898) Weighting: 50/50 telephone/online, education, region, thumb

52 © 2015 Ipsos. Online/Telephone Research (Call Poll) ELECTION DEBRIEF 2 waves consisting of minimum sample sizes of: 1/3 1/3 1/3 Ampario Allocated Reallocated 1,000 Telephone Completes  1,000 Online OMNI  roughly: 40% cellphone 3-day field window   Sample blended together to create 4 quadrants: 1) telephone landline; 2) telephone cell; 3) online panel, and 4) online non-panel - Result somewhere in the middle… and KEY BENEFITS:  Telephone poll to act as a gut check on our online  greater confidence in our figures  Triangulation among various sample sources mitigates bias of any one methodology  Allows us to formulate a weighting scheme to mitigate panel bias

53 © 2015 Ipsos. Call Poll Sample Breakdown ELECTION DEBRIEF Base: Final Call Poll (Total n=2,503; Online n=1,502; CATI n=1,001) 16% 24% 13% 47% CATI Mobile CATI Landline Online Ampario Online Panel 60% Online 40% CATI

55 © 2015 Ipsos. Lessons Learned and Next Steps ELECTION DEBRIEF Mixed mode methods show promise. Ipsos will continue to explore and share what we learn. Offset biases associated with specific modes of data collection with minimum weighting.

56 © 2015 Ipsos. Lessons Learned and Next Steps ELECTION DEBRIEF Online and offline methods consistently under and over-estimate specific demographic groups. Also, over and under-estimate specific voter mindsets. Evidence building that: …online overestimates progressive voters …telephone overestimates conservative voters.

57 © 2015 Ipsos. Lessons Learned and Next Steps ELECTION DEBRIEF LIKELY VOTER ESTIMATES, while logically attractive, work better in theory than in practice. What’s missing? How do we make them better?

59 © 2015 Ipsos. Final Weighted Call Poll by Likely Voter ELECTION DEBRIEF 38% 37% 37% 37% 37% 31% 32% 32% 33% 32% 22% 22% 22% 22% 22% 4% 2% 2% 2% 3% 4% 5% 5% 5% 4% Final 55% Turnout 60% Turnout 65% Turnout 70% Turnout Liberals Conservative NDP Green Bloc Base: Final Call Poll (Online Total Decided Voters n=1,328; I-Say Allocated n=427; I-Say Re-Allocated n=624; Ampario n=277) Weighting: 50/50 telephone/online, education, region, thumb

The Death of Polling Version 1 Public 60 Thanks for listening Q&A #ipsosmorilive