The PollyVote
Combining forecasts for
U.S. Presidential Elections
Andreas Graefe, Karlsruhe Institute of Technology
J. Sco...
Background on the PollyVote project
The PollyVote project was begun in 2003 to
demonstrate the value of forecasting princi...
Performance of the PollyVote
The PollyVote combined forecasts to obtain highly accurate
forecasts of U.S. Presidential Ele...
Power of combining
Question: What is the ratio of students per teacher in
primary schools in Romania?
Judge Estimate Error...
Procedure and conditions for combining forecasts
Procedure:
Mechanically combine forecasts equal weights
(unless you have ...
Benefits of combining
1. Improves accuracy
2. Avoids large errors
3. Provides an additional assessment of
uncertainty
4. C...
Costs of combining
1. Requires expertise with various methods
2. Higher expenses with more methods
7
Prior research
Meta-analysis of 30 studies on combining: 12% error
reduction vs. error of typical component.
Recommendatio...
9
Polly’s
Components
Polly‘s components
Polls
IEM
prediction
market
Experts
Quantitative
models
10
Polly’s
Components
Polls
Problem:
• Polls often unreliable, especially
early in campaign
• Large differences in results...
11
Polly’s
Components
IEM
prediction
market
Within component Combining
• Polly’s prediction market: Iowa Electronic
Market...
12
Polly’s
components Experts
Within component Combining
• Survey of experts
• Assumptions: Experts possess
• Information ...
13
Polly’s
components
Quantitative
models
Within combining Combining
 Models focus on 2 to 7 variables,
most often
 Incu...
14
Mean error reduction
(93 days prior to
Election Day,
1992 to 2008)
Polly’s
components
Gains from combining within compo...
Polly’s
components
Combining across components
Polls IEM Experts Models
Within components Combining Combining Combining Co...
Mean error reduction
(93 days prior to
Election Day,
1992 to 2008)
Polly’s
components
Gains from combining across componen...
Mean error reduction
(93 days prior to
Election Day,
1992 to 2008)
Polly’s
components
Gains from combining within & across...
If combining forecasts is so useful,
why is it seldom used?
18
1. Managers do not believe combining helps
In four experiments with MBAs at INSEAD, most
subjects did not realize that the...
2. Some forecasters mistakenly believe
they are combining properly
People often use unaided judgment to assign
differentia...
3. Managers, forecasters, and researchers are
persuaded by complexity
Simple models often predict complex problems better
...
4. Forecasters build reputation with extreme
forecasts
Forecasters do not want to get lost in the crowd.
More extreme fore...
5. People mistakenly believe they can
identify the most accurate forecast
In a series of experiments, when given two
estim...
Why doesn’t the PollyVote
capture mass media attention?
The PollyVote varies little and, basically, is never
wrong. Thus, ...
Accuracy problem is solved for
major elections
PollyVote deviation averaged 0.4% for the 2004
and 2008 U.S. presidential e...
Applications of combining
All organizations can benefit from combining.
References
Armstrong, J. S. (2001). Combining forecasts. In: J. S. Armstrong (Ed.),
Principles of Forecasting: A Handbook ...
Upcoming SlideShare
Loading in …5
×

Forecasting elections from voters' perceptions of candidates' ability to handle issues

315 views
274 views

Published on

Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences Perspective November 21, 2010

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
315
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Forecasting elections from voters' perceptions of candidates' ability to handle issues

  1. 1. The PollyVote Combining forecasts for U.S. Presidential Elections Andreas Graefe, Karlsruhe Institute of Technology J. Scott Armstrong, Wharton School, University of Pennsylvania Randall Jones, Jr., University of Central Oklahoma Alfred Cuzán, University of West Florida The full paper to this talk can be downloaded at: tinyurl.com/combiningelections. Bucharest Dialogues on Expert Knowledge, Prediction, Forecasting: A Social Sciences Perspective November 21, 2010
  2. 2. Background on the PollyVote project The PollyVote project was begun in 2003 to demonstrate the value of forecasting principles by applying them to election forecasting. The initial focus was on combining forecasts.
  3. 3. Performance of the PollyVote The PollyVote combined forecasts to obtain highly accurate forecasts of U.S. Presidential Election outcomes: – Prospectively for 2004 and 2008 (MAE: 0.4 percentage points) – Retrospectively for 1992 to 2000 Across these five elections, the PollyVote was on average more accurate than each of its components: - Polls - Prediction markets - Experts - Statistical models Polly achieved this without knowing anything about politics.
  4. 4. Power of combining Question: What is the ratio of students per teacher in primary schools in Romania? Judge Estimate Error 1 18 .5 2 19 1.5 Typical error of individual estimate 1 Combined estimate 18.5 1 Error reduction through combining 0% Judge Estimate Error 1 18 .5 2 16 1.5 Typical error of individual estimate 1 Combined estimate 17 0.5 Error reduction through combining 50%
  5. 5. Procedure and conditions for combining forecasts Procedure: Mechanically combine forecasts equal weights (unless you have strong evidence for differential weights) Conditions: 1. Several forecasts available 2. Uncertainty about which forecasts is most accurate (although combing is often beneficial even when the best method is known beforehand) Conditions for when combining is most beneficial: 1. Different forecasting methods are available 2. Forecasts rely upon different data
  6. 6. Benefits of combining 1. Improves accuracy 2. Avoids large errors 3. Provides an additional assessment of uncertainty 4. Can be used for nearly all forecasting problems. 5. Simple to describe and apply.
  7. 7. Costs of combining 1. Requires expertise with various methods 2. Higher expenses with more methods 7
  8. 8. Prior research Meta-analysis of 30 studies on combining: 12% error reduction vs. error of typical component. Recommendation: Combine forecasts from different methods that use different information [Armstrong, 2001] However, few studies have focused on the ex ante conditions of when combining is most beneficial. 8
  9. 9. 9 Polly’s Components Polly‘s components Polls IEM prediction market Experts Quantitative models
  10. 10. 10 Polly’s Components Polls Problem: • Polls often unreliable, especially early in campaign • Large differences in results of individual polls conducted around the same time Polls Within component Combining
  11. 11. 11 Polly’s Components IEM prediction market Within component Combining • Polly’s prediction market: Iowa Electronic Markets (IEM) • 7-day rolling average of daily market prices • Adjust for overreactions of market such as information cascades IEM prediction market
  12. 12. 12 Polly’s components Experts Within component Combining • Survey of experts • Assumptions: Experts possess • Information from polls • Knowledge about the effect of debates, campaigns, etc. Experts
  13. 13. 13 Polly’s components Quantitative models Within combining Combining  Models focus on 2 to 7 variables, most often  Incumbent‘s popularity  State of economy  Individual accuracy of models varies across elections Quantitative models
  14. 14. 14 Mean error reduction (93 days prior to Election Day, 1992 to 2008) Polly’s components Gains from combining within components Polls IEM Experts Models Within components Combining Combining Combining Combining 14% 9% 21%18%
  15. 15. Polly’s components Combining across components Polls IEM Experts Models Within components Combining Combining Combining Combining Across components Combining (unweighted average) PollyVote-Prediction
  16. 16. Mean error reduction (93 days prior to Election Day, 1992 to 2008) Polly’s components Gains from combining across components Polls (combined) IEM (combined) Experts (combined) Models (combined) PollyVote-Prediction 50% 1% 32%43%
  17. 17. Mean error reduction (93 days prior to Election Day, 1992 to 2008) Polly’s components Gains from combining within & across components Typical Poll Original IEM Typical Experts Typical Models PollyVote-Prediction 58% 10% 58%52%
  18. 18. If combining forecasts is so useful, why is it seldom used? 18
  19. 19. 1. Managers do not believe combining helps In four experiments with MBAs at INSEAD, most subjects did not realize that the error of the average forecast would be less than the error of the typical forecast. Most subjects thought that averaging forecasts would yield average performance. [Larrick & Soll, 2006] 19
  20. 20. 2. Some forecasters mistakenly believe they are combining properly People often use unaided judgment to assign differential weights to individual forecasts. Informal combining is likely to be harmful as people can select a forecast that suits their biases. 20
  21. 21. 3. Managers, forecasters, and researchers are persuaded by complexity Simple models often predict complex problems better than more complex ones. [Hogarth, in press] These findings are difficult to believe. There is a strong belief that complex models are necessary to solve complex problems. 21
  22. 22. 4. Forecasters build reputation with extreme forecasts Forecasters do not want to get lost in the crowd. More extreme forecasts usually gain more attention and the media is more likely to report them. [Batchelor, 2007]
  23. 23. 5. People mistakenly believe they can identify the most accurate forecast In a series of experiments, when given two estimates as advice, most people chose one instead of averaging them – and thereby reduced accuracy. [Soll & Larrick, 2009]
  24. 24. Why doesn’t the PollyVote capture mass media attention? The PollyVote varies little and, basically, is never wrong. Thus, no entertainment value. Instead of accuracy, voters want excitement – and hope for their candidate. 24
  25. 25. Accuracy problem is solved for major elections PollyVote deviation averaged 0.4% for the 2004 and 2008 U.S. presidential elections and substantial improvements are scheduled for 2012. Polly is available to researchers and practitioners for elections in the U.S., as well as in other countries. 25
  26. 26. Applications of combining All organizations can benefit from combining.
  27. 27. References Armstrong, J. S. (2001). Combining forecasts. In: J. S. Armstrong (Ed.), Principles of Forecasting: A Handbook for Researchers and Practitioners, Norwell: Kluwer, pp.417-439. Batchelor, R. (2007). Bias in macroeconomic forecasts, International Journal of Forecasting, 23, 189-203. Hogarth, R. (in press). When simple is hard to accept. In P. M. Todd, G. Gigerenzer, & The ABC Research Group (Eds.), Ecological rationality: Intelligence in the world. Oxford: Oxford University Press. Larrick, R. P. & Soll, J. B. (2006). Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science, 52, 111-127. Soll, J. B. & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people use others’ opinions, Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 780-805.

×