Is the Crowd wise enough to predict the Election
Results for the German “Bundestag”?
Thomas Donath, NORDLIGHT research Gmb...
Agenda


                                         01                  Check In


                                         ...
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“?   T. Donath & C. Irmer   Pforzheim, 2...
Is the Crowd wise enough to predict the Election Results for the German „Bundestag“?   T. Donath & C. Irmer   Pforzheim, 2...
What does it take to produce smarter
                            outcomes with a crowds “collective
                      ...
The Research Question



         Could a „Wisdom of Crowds“ approach to
                        election prognosis potent...
Sample

        2,323 valid respondents (X,XXX invitations -> XX% valid response rate)
                     18 years and ...
Procedure for Data Collection

             Instruction (translated from German)
                 1. We welcome each ind...
Wisdom of Crowd Result
        CDU / CSU
                                                                                 ...
Wisdom of Crowd Result
        SPD
                                                                                       ...
Wisdom of Crowd Result
        FDP
                                                                                       ...
Wisdom of Crowd Result
        Die Linke
                                                                                 ...
Wisdom of Crowd Result
        B90 – Die Grünen
                                                                          ...
Wisdom of Crowd Result
        Andere
                                                                                    ...
Comparison with Polls
                                                                                                    ...
Moderator Variables I
                                                                             sum %-point
           ...
Moderator Variables II
                                                                             sum %-point
          ...
Correlation with own Choice
                                                                                              ...
Wisdom of Crowds vs. Unweighted, non-
        represantative Poll
                                                        ...
Effect of seen Prognoses on Precision
      People with (self-claimed) awareness or knowledge of recent prognoses yield b...
Summary of Results
      Wisdom of Crowds outperforms unweighted, non-representative poll result of the same
       sampl...
Outlook
      Wisdom of Crowds might not serve to compete with existing polling methods (yet!)

           but…

      ...
THANK YOU VERY MUCH!



Thomas Donath                                                                          Christoph I...
Upcoming SlideShare
Loading in …5
×

Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?

726 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
726
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Is the Crowd wise enough to predict the Election Results for the German “Bundestag”?

  1. 1. Is the Crowd wise enough to predict the Election Results for the German “Bundestag”? Thomas Donath, NORDLIGHT research GmbH Christoph Irmer, ODC Services GmbH
  2. 2. Agenda 01 Check In 02 Research Question 03 Methods & Data 04 Results 05 Summary / Outlook Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 2
  3. 3. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 3
  4. 4. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 4
  5. 5. What does it take to produce smarter outcomes with a crowds “collective intelligence”? 01 Diversity of opinion 02 Independence of opinion 03 Decentralization 04 Aggregation Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 5
  6. 6. The Research Question Could a „Wisdom of Crowds“ approach to election prognosis potentially compete with or even outperform traditional means of prognosis Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 6
  7. 7. Sample 2,323 valid respondents (X,XXX invitations -> XX% valid response rate)  18 years and older  widely spread demographics  no other screening or weighting  recruited through the ODC Services GmbH online access panel “opinion-people” Data filtering: respondents were only excluded if  sum of % >100.0  sum of % <100.0  % for a single party = 100.0 Incentive  for the smallest summed absolute deviation in %  1. price: 300 €, 2. and 3. price 100 € each Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 7
  8. 8. Procedure for Data Collection  Instruction (translated from German)  1. We welcome each individual participation. We only ask you to submit a serious estimate.  2. Please submit your personal prognosis. To do so, please think about how the percents will be distributed across the parties.  3. Please submit a prognosis as precise as possible and make use of visual Feedback: the position after the decimal point. sum is displayed green if equal to 100.0  How many percent of votes will the following parties receive in the Bundestagswahl 2009 on next Sunday? Please do not enter the result you wish for, but your estimate what the official final result will be. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 8
  9. 9. Wisdom of Crowd Result CDU / CSU  Distribution of estimate CDU / CSU 400  survey estimate, mean = 35.0% reading help: approx. 300  election result: 33.8% individuals estimated the CDU / CDU to obtain approx. 34%   +1.2 (34.0% - 34.9%) 300  CDU / CSU overestimated  standard deviation = 5.5  95% confidence interval: 34.7 – 200 35.2  kurtosis = 6.4  leptokurtic, „supergaußförmig“ 100  skewness = -0.1  very low skewness, symmetrical distribution  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% CDU / CSU Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 9
  10. 10. Wisdom of Crowd Result SPD  Distribution of estimate SPD 400  survey estimate, mean = 27.4%  election result: 23.0%   +4.4 300  SPD strongly overestimated  standard deviation = 5.3  95% confidence interval: 27.2 – 200 27.6  kurtosis = 2.0  slightly leptokurtic, „supergaußförmig“ 100  skewness = 0.3  low skewness, fairly symmetrical distribution  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% SPD Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 10
  11. 11. Wisdom of Crowd Result FDP  Distribution of estimate FDP 400  survey estimate, mean = 12.3%  election result: 14.6%   -2.3 300  FDP strongly underestimated  standard deviation = 3.8  95% confidence interval: 12.1 – 200 12.5  kurtosis = 8.0  leptokurtic, „supergaußförmig“ 100  skewness = 1.3  slightly right- skewed  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% FDP Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 11
  12. 12. Wisdom of Crowd Result Die Linke  Distribution of estimate Die Linke 400  survey estimate, mean = 10.5%  election result: 11.9%   -1.4 300  Die Linke underestimated  standard deviation = 4.9  95% confidence interval: 10.3 – 200 10.7  kurtosis = 9.0  leptokurtic, „supergaußförmig“ 100  skewness = 1.9  slightly right- skewed  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% Die Linke Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 12
  13. 13. Wisdom of Crowd Result B90 – Die Grünen  Distribution of estimate Bündnis 90 - Die Grünen 400  survey estimate, mean = 10.0%  election result: 10.7%   -0.7 300  B90 / Die Grünen underestimated  standard deviation = 3.3 200  95% confidence interval: 9.9 – 10.2  kurtosis = 2.9  leptokurtic, „supergaußförmig“ 100  skewness = 0.3  very low skewness, fairly symmetrical distribution  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% Bündnis 90 - Die Grünen Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 13
  14. 14. Wisdom of Crowd Result Andere  Distribution of estimate Andere (other parties) 400  survey estimate, mean = 4.8%  election result: 6.0%   -1.2 300  “Andere” underestimated  standard deviation = 3.9  95% confidence interval: 4.6 – 4.9 200  kurtosis = 33.6  strongly leptokurtic, „supergaußförmig“  skewness = 4.1  right-skewed 100  n=2,323 10% 20% 30% 40% 50% 60% 70% 80% 90% Andere (other parties) Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 14
  15. 15. Comparison with Polls 35,0 36,0 33,0 CDU / CSU 35,0 35,0 35,0 33,8 27,4 25,0 25,0 SPD 24,0 25,0 larger deviation from 26,0 23,0 final result and polls 12,3 13,0 14,0 larger deviation from FDP 13,5 13,0 14,0 final result and polls 14,6 10,5 sum of absolute 11,0 12,0 %-points difference Die Linke 11,5 12,0 from final result 11,0 11,9 18.09.09 - 23.09.09 Wisdom of Crowds 11.1%-points 10,0 20.09.09 Forschungsgruppe Wahlen (FGW) 8.4%-points 10,0 Bündnis 90 – 10,0 11,0 25.09.09 Forsa 4.2%-points Die Grünen 11,0 10,0 10,7 22.09.09 IfD Allensbach 5.0%-points 4,8 5,0 18.09.09 Emnid 7.2%-points 6,0 Andere 4,0 5,0 17.09.09 Infratest dimap 8.4%-points 4,0 6,0 27.09.09 Final Result for Election Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 15
  16. 16. Moderator Variables I sum %-point deviation (group estimate n vs. final result) sex Mann-Whitney-U-Test male 9,7% (1237) Z -11,9 female 12,7% (1073) p 0,000 age Kruskal-Wallis-Test 18 - 29 years 9,4% (441) Chi-Quadrat 72,0 30 - 39 years 11,6% (418) p 0,000 40 - 49 years 12,3% (486) 50 - 59 years 11,1% (412) 60 - 69 years 11,1% (371) 70 years and older 11,6% (142) graduation Kruskal-Wallis-Test Hauptschulabschluss (Volksschulabschluss) 12,2% (499) Chi-Quadrat 91,5 Realschule (Mittlere Reife) 11,6% (676) p 0,000 Abschluss der Polytechnischen Oberschule 10,8% (127) Fachhochschulreife 10,5% (228) Abitur (allgemeine oder fachgebundene Hochschulreife) 10,3% (686) How often do you inform yourself about the political Kruskal-Wallis-Test occurances? to be honest, not at all 14,2% (171) Chi-Quadrat 235,1 less than once per month 12,1% (189) p 0,000 once per month 16,0% (162) once per week 11,7% (465) several times per week 9,6% (1212)  Virtually all socio- and psychographic moderator variables influence the precision of Wisdom of Crowds estimates. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 16
  17. 17. Moderator Variables II sum %-point deviation (group estimate n vs. final result) How interested are you in politics? Kruskal-Wallis-Test to be honest, I am not interested in politics at all 14,4% (202) Chi-Quadrat 247,7 I am interested a tiny bit in politics. 12,9% (405) p 0,000 I am interested somewhat in politics. 11,6% (915) I am strongly interested in politics. 9,5% (595) I am very strongly interested in politics. 8,8% (191) Which statement fits you personally the most? Kruskal-Wallis-Test I have already voted via letter vote. 10,3% (381) Chi-Quadrat 75,1 I am going to vote and know exactly which party I will vote 10,8% (1167) p 0,000 for with my Zweitstimme (Landesliste). I am going to vote, but I do not yet know exactly which 11,2% (405) party I will vote for with my Zweitstimme (Landesliste). I do not know yet whether I will vote, but I would know exactly which party I would vote for with my Zweitstimme 13,2% (94) (Landesliste). I do not know yet whether I will vote and also I would not know exactly which party I would vote for with my 12,3% (115) Zweitstimme (Landesliste). I am definitely not going to vote. 13,0% (105) Have you seen prognoses for the results of the Kruskal-Wallis-Test Bundestagswahl 2009 within the last weeks? yes, and I can remember (roughly) the results 10,1% (597) Chi-Quadrat 221,9 yes, but I cannot remember exactly the results 11,0% (961) p 0,000 no, I have not seen any prognoses 12,8% (700) other sign. moderator variable: state (Bundesland), household size, marital status, employment status, type of work, income Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 17
  18. 18. Correlation with own Choice personal estimate for final result Korrelati onen Bündnis 90 - CDU / CSU SPD FDP Die Linke Die Grünen Andere Kendall-Tau-b Sonntagsf rage: CDU / Korrelationskoef f izient ,221** -,072** ,094** -,087** -,058** -,051** CSU Sig. (2-seitig) ,000 ,000 ,000 ,000 ,001 ,003 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: SPD Korrelationskoef f izient -,066** ,166** -,107** -,023 ,043* -,035* own choice for vote Sig. (2-seitig) ,000 ,000 ,000 ,181 ,013 ,044 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: FDP Korrelationskoef f izient ,018 -,081** ,164** -,028 ,008 -,020 Sig. (2-seitig) ,297 ,000 ,000 ,099 ,645 ,251 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: DIE Korrelationskoef f izient -,125** -,075** -,067** ,236** -,012 ,003 LINKE Sig. (2-seitig) ,000 ,000 ,000 ,000 ,488 ,867 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: BÜNDNIS Korrelationskoef f izient -,048** ,036* -,044** -,027 ,145** -,029 90/DIE GRÜNEN Sig. (2-seitig) ,005 ,034 ,010 ,107 ,000 ,092 N 2323 2323 2323 2323 2323 2323 Sonntagsf rage: Andere Korrelationskoef f izient -,085** -,019 -,033 ,027 -,036* ,127** Sig. (2-seitig) ,000 ,270 ,053 ,120 ,037 ,000 N 2323 2323 2323 2323 2323 2323 **. Die Korrelation ist auf dem 0,01 Niv eau signif ikant (zweiseitig). *. Die Korrelation ist auf dem 0,05 Niv eau signif ikant (zweiseitig).  There is an opinion bias: Respondents tend to estimate the final result higher for those parties they would also vote for.  The effect is on average 2,7%-points.  I.e. people estimate the final result for parties they would vote for 2,7%-points higher compared to people who would vote for other parties. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 18
  19. 19. Wisdom of Crowds vs. Unweighted, non- represantative Poll 28,1 CDU / CSU 35,0 33,8 22,1 SPD 27,4 23,0 13,5 FDP 12,3 14,6 14,6 DIE LINKE 10,5 11,9 sum of absolute BÜNDNIS 90/DIE 10,0 %-points deviation 10,0 from final result GRÜNEN 10,7 11,7 Sonntagsfrage (Votes for the Panel) 16.8%-points Andere 4,8 Wisdom of Crowds (Estimate of the Panel) 11.1%-points 6,0 Final Result for Election  The Wisdom of Crowds method yields a better estimate for the outcome of the election than a non- weighted, non-representative poll (Sonntagsfrage) from the same panel would have yielded.  The “subjective” opinion bias – overestimating the preferred parties outcome – is clearly weaker than the “objective” reasoning (see esp. CDU/CSU, SPD, Andere). Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 19
  20. 20. Effect of seen Prognoses on Precision  People with (self-claimed) awareness or knowledge of recent prognoses yield better estimates.  However, it can be shown that Wisdom of Crowds also works in absence of such knowledge.  Political interest in itself – even without such knowledge – increases the precision.  Result: People who claim to have seen recent prognoses submit better estimates.  Hypothesis: Without representative polls in media, even people interested in politics would be unable to estimate the outcome precisely.  However, there is a confoundation of seeing prognoses with general interest, the latter also being a strong moderator variable for estimate precision. (Kendall Tau- b=0.390, p=0.000, n=2,244, fairly linear function for the used scales)  Partial Correlation (functions of inter-correlation for used scales are all fairly linear)  having seen prognoses with estimate precision (controlled for interest): r=0.172, p=0.000, n=2,241, r2=2.9%  interest with estimate precision (controlled for having seen prognoses): r=0.220, p=0.000, n=2,241, r 2=4.9%  Group 1: no prognoses seen, slight to high interest in politics difference = 11.3 %-points  Group 2: no prognoses seen, low interest difference = 14.7 %-points  (prognoses seen and roughly remembered difference = 10.1 %-points)  n=389; n=307  Mann-Whitney-U-Test: Z=-6.2, p=0.000  Hypothesis can be rejected, based on self-report of participants: even without seeing recent prognoses, participants who are interested in politics provide good estimates.  Of course there is no reasonable way to truly eliminate seeing or benefiting from past prognoses seen by participants. However, this is not a requirement of the Wisdom of Crowds approach. It actually requires to some extend informed participants. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 20
  21. 21. Summary of Results  Wisdom of Crowds outperforms unweighted, non-representative poll result of the same sample.  In this single study however, Wisdom of Crowds is inferior compared to representative polls.  The self-reported knowledge of recent prognoses drastically increases the quality of the prediction. But also people interested in politics without that knowledge yield fairly accurate predictions drastically outperforming participants without interest.  Generally prediction quality is influenced massively by moderator variables.  This fact is somewhat contrary to the claim by Surowiecki that the individual knowledge of people hardly matters for the precision. Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 21
  22. 22. Outlook  Wisdom of Crowds might not serve to compete with existing polling methods (yet!)  but…  imagine it would not have been election polls but e.g. Product-Tests, Ad-Tests, … : 35 CDU/CSU Product C 33,8 27,4 SPD Product D 23 FDP Product A 12,3 14,6 Wisdom Of Crowds Linke Final Result Product F 10,5 11,9 B90 / Grüne Product E 10 10,7 Andere Product B 4,8 6 Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 22
  23. 23. THANK YOU VERY MUCH! Thomas Donath Christoph Irmer NORDLIGHT research GmbH ODC Services GmbH t.donath@nordlight-research.com ci@odc-services.com Is the Crowd wise enough to predict the Election Results for the German „Bundestag“? T. Donath & C. Irmer Pforzheim, 28.05.2010 23

×