Training deck for rolling forecasts in the Good Judgment Project's continuous elicitation condition

1. P I C K I N G T H E R I G H T D A T E S A N D Q U A N T I T I E S
Rolling Forecasts

2. Forecasting isn’t just about saying yes/no
 We also care about timing and
amounts!
 By what date will Bashar al-Assad
vacate the office of president of
Syria?
 How many Japanese nuclear
reactors will be operational by
June 1, 2014? (This question
appeared last year.)

3. Thinking about Uncertainty
 You can’t definitively know the
answers in advance, but some
quantities and dates are more
likely than others.
 Depending on what you know or
learn about an event, you can
weight one option more than
another.
 The more you know, the more you
can assign higher probabilities to
options, the less you know, the
broader the possible range can be.

4. Dealing with Uncertainty
 People deal with uncertainty all the time:
 How long will my commute be?
 How much will my vacation cost?
 What rate of return can I expect from my portfolio?
The goals of this training are to:
 Introduce you to the continuous estimation (CE)
format of forecasting problems.
 Teach you to express uncertainty in quantitative
terms and think about uncertainty in politics.
 Explain how your forecasts will be scored.

5. Question Format
 Every forecasting problem will be posed in the form of non-
overlapping intervals that cover all possibilities.
For example:
 When will an asteroid strike in Nauru?
Before October 26, 2014
Between October 26, 2014 and December 2014
Between December 30, 2014 and March 3, 2015
Between March 4, 2015 and May 8, 2015
Not before May 8, 2015

6. Answer Format
 You have a total of 100 tokens to assign across all bins.
 Each token represents 1% of your belief that the “bin” is the right one.
 Put tokens into each bin until you’ve exhausted your supply.

7. Rolling Bins
 When a bin expires, it will be deleted (-), and the last
bin will be split (+) as to keep the same number of bins
for the question.

8. Tips on forecasting dates and quantities
 Hunt* for the right information- any clues as to what might
happen and how similar events have resolved in the past.
 Think about plausible scenarios that can lead to outcomes
within each of the listed intervals.
 Beware of overconfidence in some outcomes versus others.
Why might your favored outcomes not occur?
 Update your forecasts as you become aware of new
information.
*Remember: CHAMPS KNOW!

9. How you’ll be scored
 Higher scores mean more
accurate forecasts.
You will receive a score
when the first bin expires.
This score will be updated
each time a bin expires.

10. Scoring
 In order to determine your bin
score, we calculate a score for every
day that that bin is open.
 If you do not forecast right away,
you will receive the group’s
median score every day until you
make your first forecast.
 Once you make a forecast, your
score on that forecast will carry
forward until you update your
forecast.

11. Scoring
We assign scores using Brier Scores.
 A Brier score is the squared deviation between your probability
and reality.
 I give 60% probability to this bin on January 3. The event
happens during the time period represented by this bin (1).
 2(0.6-1)² = 0.32 is your Brier score.
 If the event didn’t happen (0) we’d see 2(0.6-0)² = 0.72.
 We then subtract your Brier Score from 2 to determine your
Daily Score. (2 – 0.32 = 1.68)

12. Scoring
 To determine a bin’s score we add the daily score
of every day inside that bin.

13. Scoring
 Every time a bin closes and
a new bin is added, we will
create a new set of bins.
 When a bin expires, its
score is added to all
existing sets. In other
words:
 When Bin 3 closes, Bin
3’s score is added to Set
A, Set B, and Set C.
 After the scores are
added to the existing
bins, Set 4 is created.
 Bin 1’s score is only
added to Set 1 because no
sets exist until Set 1
closes.

14. Scoring
 When the question closes,
all open bins are closed
and scored.
 Green scores are calculated
after a question closes.
 The total scores of each
Set are added together to
form your final score.

15. Raising Your Score
 You are STRONGLY encouraged to UPDATE
YOUR FORECAST as often as possible.
 It is particularly important to update EVERY TIME
A BIN CLOSES to get a good score.
 Our scoring method rewards frequent updates
by taking your score on every day into account.
 Frequent updates mean more chances to raise
your score!

16. Scoring Summary
 In short, when a question closes we will add
the score of every set of bins that you
forecast on together to determine your final
score.
The longer a question is open, the higher your
score can be.
It is in your best interests to update as often
as possible.

17. Leaderboard
 During the season, everyone
will be listed by their screen
name and rank, ordered by
their mean scores for all
closed questions.
 Do well and find yourself at
the top of the leaderboard!