Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?

Is the Crowd wise enough to predict the Election Results for the German “Bundestag”? Thomas Donath, NORDLIGHT research GmbH Christoph Irmer, ODC Services GmbH

Agenda 01 Check In 02 Research Question 03 Methods & Data 04 Results 05 Summary / Outlook Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 2

Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 3

Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 4

What does it take to produce smarter outcomes with a crowds “collective intelligence”? 01 Diversity of opinion 02 Independence of opinion 03 Decentralization 04 Aggregation Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 5

The Research Question Could a „Wisdom of Crowds“ approach to election prognosis potentially compete with or even outperform traditional means of prognosis Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 6

Sample 2,323 valid respondents (X,XXX invitations -> XX% valid response rate)  18 years and older  widely spread demographics  no other screening or weighting  recruited through the ODC Services GmbH online access panel “opinion-people” Data filtering: respondents were only excluded if  sum of % >100.0  sum of % <100.0  % for a single party = 100.0 Incentive  for the smallest summed absolute deviation in %  1. price: 300 €, 2. and 3. price 100 € each Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 7

Procedure for Data Collection  Instruction (translated from German)  1. We welcome each individual participation. We only ask you to submit a serious estimate.  2. Please submit your personal prognosis. To do so, please think about how the percents will be distributed across the parties.  3. Please submit a prognosis as precise as possible and make use of visual Feedback: the position after the decimal point. sum is displayed green if equal to 100.0  How many percent of votes will the following parties receive in the Bundestagswahl 2009 on next Sunday? Please do not enter the result you wish for, but your estimate what the official final result will be. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 8

Wisdom of Crowd Result CDU / CSU  Distribution of estimate CDU / CSU 400  survey estimate, mean = 35.0% reading help: approx. 300  election result: 33.8% individuals estimated the CDU / CDU to obtain approx. 34%   +1.2 (34.0% - 34.9%) 300  CDU / CSU overestimated  standard deviation = 5.5  95% confidence interval: 34.7 – 200 35.2  kurtosis = 6.4  leptokurtic, „supergaußförmig“ 100  skewness = -0.1  very low skewness, symmetrical distribution  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% CDU / CSU Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 9

Wisdom of Crowd Result SPD  Distribution of estimate SPD 400  survey estimate, mean = 27.4%  election result: 23.0%   +4.4 300  SPD strongly overestimated  standard deviation = 5.3  95% confidence interval: 27.2 – 200 27.6  kurtosis = 2.0  slightly leptokurtic, „supergaußförmig“ 100  skewness = 0.3  low skewness, fairly symmetrical distribution  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% SPD Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 10

Wisdom of Crowd Result FDP  Distribution of estimate FDP 400  survey estimate, mean = 12.3%  election result: 14.6%   -2.3 300  FDP strongly underestimated  standard deviation = 3.8  95% confidence interval: 12.1 – 200 12.5  kurtosis = 8.0  leptokurtic, „supergaußförmig“ 100  skewness = 1.3  slightly right- skewed  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% FDP Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 11

Wisdom of Crowd Result Die Linke  Distribution of estimate Die Linke 400  survey estimate, mean = 10.5%  election result: 11.9%   -1.4 300  Die Linke underestimated  standard deviation = 4.9  95% confidence interval: 10.3 – 200 10.7  kurtosis = 9.0  leptokurtic, „supergaußförmig“ 100  skewness = 1.9  slightly right- skewed  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% Die Linke Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 12

Wisdom of Crowd Result B90 – Die Grünen  Distribution of estimate Bündnis 90 - Die Grünen 400  survey estimate, mean = 10.0%  election result: 10.7%   -0.7 300  B90 / Die Grünen underestimated  standard deviation = 3.3 200  95% confidence interval: 9.9 – 10.2  kurtosis = 2.9  leptokurtic, „supergaußförmig“ 100  skewness = 0.3  very low skewness, fairly symmetrical distribution  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% Bündnis 90 - Die Grünen Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 13

Wisdom of Crowd Result Andere  Distribution of estimate Andere (other parties) 400  survey estimate, mean = 4.8%  election result: 6.0%   -1.2 300  “Andere” underestimated  standard deviation = 3.9  95% confidence interval: 4.6 – 4.9 200  kurtosis = 33.6  strongly leptokurtic, „supergaußförmig“  skewness = 4.1  right-skewed 100  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% Andere (other parties) Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 14

Comparison with Polls 35,0 36,0 33,0 CDU / CSU 35,0 35,0 35,0 33,8 27,4 25,0 25,0 SPD 24,0 25,0 larger deviation from 26,0 23,0 final result and polls 12,3 13,0 14,0 larger deviation from FDP 13,5 13,0 14,0 final result and polls 14,6 10,5 sum of absolute 11,0 12,0 %-points difference Die Linke 11,5 12,0 from final result 11,0 11,9 18.09.09 - 23.09.09 Wisdom of Crowds 11.1%-points 10,0 20.09.09 Forschungsgruppe Wahlen (FGW) 8.4%-points 10,0 Bündnis 90 – 10,0 11,0 25.09.09 Forsa 4.2%-points Die Grünen 11,0 10,0 10,7 22.09.09 IfD Allensbach 5.0%-points 4,8 5,0 18.09.09 Emnid 7.2%-points 6,0 Andere 4,0 5,0 17.09.09 Infratest dimap 8.4%-points 4,0 6,0 27.09.09 Final Result for Election Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 15

Moderator Variables I sum %-point deviation (group estimate n vs. final result) sex Mann-Whitney-U-Test male 9,7% (1237) Z -11,9 female 12,7% (1073) p 0,000 age Kruskal-Wallis-Test 18 - 29 years 9,4% (441) Chi-Quadrat 72,0 30 - 39 years 11,6% (418) p 0,000 40 - 49 years 12,3% (486) 50 - 59 years 11,1% (412) 60 - 69 years 11,1% (371) 70 years and older 11,6% (142) graduation Kruskal-Wallis-Test Hauptschulabschluss (Volksschulabschluss) 12,2% (499) Chi-Quadrat 91,5 Realschule (Mittlere Reife) 11,6% (676) p 0,000 Abschluss der Polytechnischen Oberschule 10,8% (127) Fachhochschulreife 10,5% (228) Abitur (allgemeine oder fachgebundene Hochschulreife) 10,3% (686) How often do you inform yourself about the political Kruskal-Wallis-Test occurances? to be honest, not at all 14,2% (171) Chi-Quadrat 235,1 less than once per month 12,1% (189) p 0,000 once per month 16,0% (162) once per week 11,7% (465) several times per week 9,6% (1212)  Virtually all socio- and psychographic moderator variables influence the precision of Wisdom of Crowds estimates. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 16

Moderator Variables II sum %-point deviation (group estimate n vs. final result) How interested are you in politics? Kruskal-Wallis-Test to be honest, I am not interested in politics at all 14,4% (202) Chi-Quadrat 247,7 I am interested a tiny bit in politics. 12,9% (405) p 0,000 I am interested somewhat in politics. 11,6% (915) I am strongly interested in politics. 9,5% (595) I am very strongly interested in politics. 8,8% (191) Which statement fits you personally the most? Kruskal-Wallis-Test I have already voted via letter vote. 10,3% (381) Chi-Quadrat 75,1 I am going to vote and know exactly which party I will vote 10,8% (1167) p 0,000 for with my Zweitstimme (Landesliste). I am going to vote, but I do not yet know exactly which 11,2% (405) party I will vote for with my Zweitstimme (Landesliste). I do not know yet whether I will vote, but I would know exactly which party I would vote for with my Zweitstimme 13,2% (94) (Landesliste). I do not know yet whether I will vote and also I would not know exactly which party I would vote for with my 12,3% (115) Zweitstimme (Landesliste). I am definitely not going to vote. 13,0% (105) Have you seen prognoses for the results of the Kruskal-Wallis-Test Bundestagswahl 2009 within the last weeks? yes, and I can remember (roughly) the results 10,1% (597) Chi-Quadrat 221,9 yes, but I cannot remember exactly the results 11,0% (961) p 0,000 no, I have not seen any prognoses 12,8% (700) other sign. moderator variable: state (Bundesland), household size, marital status, employment status, type of work, income Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 17

Correlation with own Choice personal estimate for final result Korrelati onen Bündnis 90 - CDU / CSU SPD FDP Die Linke Die Grünen Andere Kendall-Tau-b Sonntagsf rage: CDU / Korrelationskoef f izient ,221** -,072** ,094** -,087** -,058** -,051** CSU Sig. (2-seitig) ,000 ,000 ,000 ,000 ,001 ,003 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: SPD Korrelationskoef f izient -,066** ,166** -,107** -,023 ,043* -,035* own choice for vote Sig. (2-seitig) ,000 ,000 ,000 ,181 ,013 ,044 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: FDP Korrelationskoef f izient ,018 -,081** ,164** -,028 ,008 -,020 Sig. (2-seitig) ,297 ,000 ,000 ,099 ,645 ,251 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: DIE Korrelationskoef f izient -,125** -,075** -,067** ,236** -,012 ,003 LINKE Sig. (2-seitig) ,000 ,000 ,000 ,000 ,488 ,867 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: BÜNDNIS Korrelationskoef f izient -,048** ,036* -,044** -,027 ,145** -,029 90/DIE GRÜNEN Sig. (2-seitig) ,005 ,034 ,010 ,107 ,000 ,092 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: Andere Korrelationskoef f izient -,085** -,019 -,033 ,027 -,036* ,127** Sig. (2-seitig) ,000 ,270 ,053 ,120 ,037 ,000 N 2323 2323 2323 2323 2323 2323 **. Die Korrelation ist auf dem 0,01 Niv eau signif ikant (zweiseitig). *. Die Korrelation ist auf dem 0,05 Niv eau signif ikant (zweiseitig).  There is an opinion bias: Respondents tend to estimate the final result higher for those parties they would also vote for.  The effect is on average 2,7%-points.  I.e. people estimate the final result for parties they would vote for 2,7%-points higher compared to people who would vote for other parties. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 18

Wisdom of Crowds vs. Unweighted, non- represantative Poll 28,1 CDU / CSU 35,0 33,8 22,1 SPD 27,4 23,0 13,5 FDP 12,3 14,6 14,6 DIE LINKE 10,5 11,9 sum of absolute BÜNDNIS 90/DIE 10,0 %-points deviation 10,0 from final result GRÜNEN 10,7 11,7 Sonntagsfrage (Votes for the Panel) 16.8%-points Andere 4,8 Wisdom of Crowds (Estimate of the Panel) 11.1%-points 6,0 Final Result for Election  The Wisdom of Crowds method yields a better estimate for the outcome of the election than a non- weighted, non-representative poll (Sonntagsfrage) from the same panel would have yielded.  The “subjective” opinion bias – overestimating the preferred parties outcome – is clearly weaker than the “objective” reasoning (see esp. CDU/CSU, SPD, Andere). Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 19

Effect of seen Prognoses on Precision  People with (self-claimed) awareness or knowledge of recent prognoses yield better estimates.  However, it can be shown that Wisdom of Crowds also works in absence of such knowledge.  Political interest in itself – even without such knowledge – increases the precision.  Result: People who claim to have seen recent prognoses submit better estimates.  Hypothesis: Without representative polls in media, even people interested in politics would be unable to estimate the outcome precisely.  However, there is a confoundation of seeing prognoses with general interest, the latter also being a strong moderator variable for estimate precision. (Kendall Tau- b=0.390, p=0.000, n=2,244, fairly linear function for the used scales)  Partial Correlation (functions of inter-correlation for used scales are all fairly linear)  having seen prognoses with estimate precision (controlled for interest): r=0.172, p=0.000, n=2,241, r2=2.9%  interest with estimate precision (controlled for having seen prognoses): r=0.220, p=0.000, n=2,241, r 2=4.9%  Group 1: no prognoses seen, slight to high interest in politics difference = 11.3 %-points  Group 2: no prognoses seen, low interest difference = 14.7 %-points  (prognoses seen and roughly remembered difference = 10.1 %-points)  n=389; n=307  Mann-Whitney-U-Test: Z=-6.2, p=0.000  Hypothesis can be rejected, based on self-report of participants: even without seeing recent prognoses, participants who are interested in politics provide good estimates.  Of course there is no reasonable way to truly eliminate seeing or benefiting from past prognoses seen by participants. However, this is not a requirement of the Wisdom of Crowds approach. It actually requires to some extend informed participants. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 20

Summary of Results  Wisdom of Crowds outperforms unweighted, non-representative poll result of the same sample.  In this single study however, Wisdom of Crowds is inferior compared to representative polls.  The self-reported knowledge of recent prognoses drastically increases the quality of the prediction. But also people interested in politics without that knowledge yield fairly accurate predictions drastically outperforming participants without interest.  Generally prediction quality is influenced massively by moderator variables.  This fact is somewhat contrary to the claim by Surowiecki that the individual knowledge of people hardly matters for the precision. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 21

Outlook  Wisdom of Crowds might not serve to compete with existing polling methods (yet!)  but…  imagine it would not have been election polls but e.g. Product-Tests, Ad-Tests, … : 35 CDU/CSU Product C 33,8 27,4 SPD Product D 23 FDP Product A 12,3 14,6 Wisdom Of Crowds Linke Final Result Product F 10,5 11,9 B90 / Grüne Product E 10 10,7 Andere Product B 4,8 6 Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 22

THANK YOU VERY MUCH! Thomas Donath Christoph Irmer NORDLIGHT research GmbH ODC Services GmbH t.donath@nordlight-research.com ci@odc-services.com Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 23

Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?

by Christoph Irmer on Jun 08, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 10

Statistics

Is the Crowd wise enough to predict the Election Results for the German “Bundestag”? — Presentation Transcript

http://www.odc-services.com	8
http://www.slideshare.net	2