This document summarizes research from multiple prediction experiments involving crowds. Some key findings include: - Crowds can make remarkably accurate predictions if individual judgments are independent and unbiased, though biases can negatively impact accuracy - Small prediction markets often outperform traditional surveys, as they weight predictions by confidence - Experts are only slightly better predictors than chance, while factoring in confidence improves accuracy - Predicting others' behaviors is easier than predicting one's own due to cognitive biases around self-knowledge

1. PREDICTING THE FUTURE
PRIMARY RESEARCH EXPLORING THE SCIENCE OF PREDICTION

2. SO WHO IS THE BEST PREDICTOR IN THE
AUDIENCE?
PRIZE: A REAL CRYSTAL BALL
BOTTLE OF CHAMPAGNE
PART 1:
MAKES SOME PREDICTIONS
PART 2:
LIVE EXPERIMENT

3. WE ASKED A GROUP OF 400 UK PANELIST TO PREDICT THE
SELLING PRICE OF THE NEW* IPAD MINI 2 WEEKS PRIOR TO ITS
LAUNCH HOW CLOSE DID THEY GET?
*Note we told them the existing selling price of the old model
WITHIN 10%, WITHIN 5%, WITHIN 3%, WITHIN 1%

4. PREDICT THE PRICE OF 100ML OF CHANEL
PERFUME?
Price in €

5. HEADS OR TAILS
PREDICT HOW MANY
SAID HEADS?

6. WHAT PROPORTION OF WINE DRINKERS IN THE UK
PREFER RED WINE?
BASED ON A POLL OF 400 WINE DRINKERS IN UK WHO WERE ASKED IF THEY PREFER RED OR WHITE
WINE

7. PREDICT IF IT WILL RAIN NEXT MONDAY

8. PREDICT HOW MANY RESEARCHERS CHECK
THEIR EMAIL BEFORE BREAKFAST?
BASED ON POLL OF ATTENDEES AT ESOMAR CONGRESS

9. DO YOU CHECK YOUR EMAILS BEFORE
BREAKFAST?

10. PREDICT WHAT MARGIN OF VICTORY OUR UK
PANELISTS PREDICTED FOR THESE 2 FOOTBALL
MATCHES
England v Montenegro
+3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland
+3 +2 +1 0 -1 -2 -3

11. WILL THE MARKET RESEARCH INDUSTRY BE
BIGGER OR SMALLER IN 10 YEARS TIME?

12. The CXO Advisory group
gathered 6,582 buy or sell
predictions from 68 different
investing gurus made between
1998 and 2012, and tracked the
results of those predictions. How
accurate were they?
WHAT % WERE CORRECT?

13. SWAP YOUR QUIZ SHEET WITH THE PERSON
NEXT TO YOU READY FOR MARKING

14. BACKGROUND
Gamification  More prediction protocols in surveys 
Fostered an interest in the science of prediction  Led to a
series of dedicated prediction experiments  Exploration of
the world of prediction market trading  Prediki
30+ Primary research experiments
500+ Predictions analysed
60+ Prediction markets v traditional research comparisons

15. PREDIKI

16. THE TYPES OF EXPERIMENTS WE HAVE RUN
• Betting on the future of brands
• Predicting why people buy things
• Predicting the behaviour of other people
• Predicting the price of things
• Predicting the election prospects of political parties
• Predicting football match results
• Predicting the outcomes of TV game shows
• Predicting the success of adverts
• Predicting future sales of products
• Predicting the future more generally

17. SO WHAT HAVE WE
LEANT ABOUT PREDICTION?

18. WHAT ARE WE GOOD AT PREDICTING AS INDIVUALS?
20%
42% 43%
32% 30%
38%
11%
21% 21% 22% 22%
29%
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Consumer
purchasing
estimates
Observed
behaviour of
others
Observed
opinion
Forecast Price
prediction
Guesswork
Correct prediction Random chance

19. SOME OF US ARE BETTER AT PREDICTING
180%
160%
140%
120%
100%
80%
60%
40%
20%
0%
Index of Prediction performance over 7 waves of experiments
score 1 score 2 score 3 score 4 score 5 score 6 score 7
top 100
bottom 100

20. 푃푟푒푑푖푐푡푖표푛 푄푢푎푙푖푡푦
=
퐼푛푓표푟푚푎푡푖표푛 × 퐸푓푓표푟푡 × 푂푏푗푒푐푡푖푣푖푡푦 × 1 − 퐷푖푓푓푖푐푢푙푡푦 × 푅푎푛푑표푚푛푒푠푠
Note: Not directly dependent on sample size
Nate Silver: Correctly predicted the outcome of all 52 states in the 2012 UK election

21. LESS ABOUT SAMPLE SIZE MORE ABOUT
SAMPLE DIVERSITY & INTELLIGENCE
Jed Christianson, University of Birmingham calculates
16 IS A CROWD

22. ..AND HOW YOU AGGREGATE
CROWD WISDOM
MEAN, MODE, MEDIAN V TRADING & DOUBLE
AUCTION TRADING

23. WHAT WE KNOW
THE WISDOM OF CROWDS
1906 Plymouth County fair
Actual weight = 1198 lb
Median average guess = 1207lb
Error = <1%

24. CROWD WISDOM IS BASED ON FILTERING
THE SIGNAL FROM THE NOISE
Each person’s prediction is made up of 2 components: information & error.
If each individual’s judgement is independent & unbiased then the error
will largely cancel itself out and the aggregation process then distils off the
inherent knowledge.

25. WHAT WE KNOW
THE WISDOM OF CROWDS
Actual selling price = £319
Median average guess = £316
Error = 1%
SCORE: 1 POINT
2013 GMI online sample

26. WITHOUT COLLECTIVE KNOWLEDGE
CROWDS CAN BE PLAIN IGNORANT

27. AN UNWISE CROWD
2014 GMI online sample
Actual weight = 550kg
Median average guess = 350kg
Error = 36%

28. CROWD WISDOM CAN BE A BIT BEHIND THE TIMES
-44%
-34% -33%
-30% -28% -26% -25% -25%
-17%
-14%
2%
5%
9% 9%
21% 23%
30%
20%
10%
0%
-10%
-20%
-30%
-40%
-50%
Examples of price predicition errors
Average 9% price lag
Price prediciton accuracy
94%
87%
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Women Men
Men less price savvy

29. YOUR PREDICTIONS
Crowd €80
Actual €120
SCORE: +/-kr100 2 POINTS
+/-kr200 1 POINT

30. “If each individual’s judgement is independent &
unbiased then the error will largely cancel itself out”
THE PROCESS OF MAKING PREDICTIONS
IS LITTERED WITH COGNITIVE BIASES

31. YOUR PREDICTIONS
68% heads
DUE TO ORDER BIAS
SCORE: +/-5% 2 POINT
+/-10% 1 POINT

32. WITH NO INFORMATION
TO GUIDE US TINY NUDGES
CAN HAVE BIG EFFECTS
ON OUR PREDICTIONS

33. HOW THIS EFFECT CORRUPTS PREDICTIONS….
What percentage of people do you think prefer white wine?
What percentage of people do you think prefer red wine?
54%
46% 46%
54%
60%
50%
40%
30%
20%
10%
0%
White Red
V
20% SHIFT IN PREDICTION

34. PREDICT IF IT WILL RAIN NEXT THURSDAY

35. Predict the chances of it raining 5 days in advance
IF RAINING TODAY +20°%
Score If you predicted correctly = 1 point

36. STUDYING THE IMPACT OF NUDGE EFFECTS:
THE INFLUENCE ONE PERSON’S OPINION HAS ON ANOTHER
2%
6%
11%
15%
20%
Personal
preferences
Self evident
predictions (e.g.
ad evaluation)
Factual (requiring
knowledge)
Inverted personal
preference (e.g.
pedicting relative
levels of dislike)
Complex
estimates
Nudge influence by prediction task
THE LESS CERTAIN PEOPLE ARE AND THE HARDER THE PREDICTION,
THE MORE WE RELY ON OTHER PEOPLE’S OPINIONS
Source: GMI research 2014

37. IN THE ABSENCE OF
KNOWLEDGE WE
PREDICT THE MAJORITY
OF OTHER PEOPLE WILL
DO & THINK THE SAME
AS US

38. 100
90
80
70
60
50
40
30
20
10
-
WHAT WILL THE WORLD BE LIKE IN 2050?
Believers
Believe this How many people believe this
55% 68%
100
90
80
70
60
50
40
30
20
10
-
Non believers
Don’t believe How many people don't believe this
those holding minority opinions assume more people agree with
them than those holding the majority opinion: is this the definition of
delusion?`

39. YOUR PREDICTIONS
SCORE: +/-5% 2 POINT
+/-10% 1 POINT
Prediction of how many other
people check emails before
50%
20%
I check my emails
before breakfast
I don't check my
emails before
breakfast
breakfast*
*Source: office poll! 80% OF MARKET RESEARCHERS

40. OUR EMOTIONS REALLY CAN
DOMINATE & BADLY DISTORT OUR
PREDICTIONS

41. PREDICT WHAT SCORES OUR UK PANELISTS PREDICTED!
England v Montenegro
+3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland
+3 +2 +1 0 -1 -2 -3
SCORE: CORRECT = 1 POINT PER QUESTION

42. 60%
50%
40%
30%
20%
10%
0%
FOOTBALL SCORE PREDICTIONS
Newcastle Liverpool
new by 3 new by 2 new by 1 draw liv by 1 liv by 2 liv by 2
50%
40%
30%
20%
10%
0%
Chelsea Cardiff
chel by 3 chel by 2 chel by 1 draw car by 1 car by 2 car by 3
50%
40%
30%
20%
10%
0%
man by 3 man by 2 man by 1 draw south by
1
south by
2
south by
3
Southhampton Man U
50%
40%
30%
20%
10%
0%
man by 3man by 2man by 1 draw south by
1
south by
2
south by
3
Southhampton Man U

43. PREDICT WHO WILL FORM THE NEXT UK GOVERNMENT: BY PARTY AFFILIATION
80%
70%
60%
50%
40%
30%
20%
10%
0%
Conservative
Government
Labour Pary
Government
Liberal Democrats
Government
Conservative &
Liberal Coalition
Labour and Liberal
Coalition
Conservative,
Liberal, & UKIP
Coalition
Conservatives Labour LiberalDemocrats UKIP

44. DIFFERENCES BETWEEN
PREDICTING WHAT OTHERS WILL DO
V
WHAT I WILL DO

45. SOCIAL COGNITIVE BIASES RENDER
PREDICTIONS ABOUT OUR OWN
BEHAVIOUR PARTICULARLY DIFFICULT
Will you tidy up after the meeting?
Yes = 50%
Predict how many will tidy up?
= 15%
TIDIED UP =13%

47. THEREFORE WE WOULD ADVOCATE A
STEREOSCOPIC APPROACH
PERSONAL PREDICTIVE

48. WE ARE OFTEN TOO TIED UP IN THE
DETAIL TO SEE THE BIGGER PICTURE
WILL THE MARRIAGE LAST?
Yes/No
Parents much better than the married
couples at predicting this
Source: Queens University Canada

49. UNABLE TO SEPARATE THE SIGNAL
FROM THE NOISE
Will the Market Research industry be bigger
or smaller in 10 years time?
Yes/No?

50. 1
0.98
0.96
0.94
0.92
0.9
Future Predictions accuracy
Experts Dillettantes
(non experts)
Chimps
(random
guesses)
CALIBRATION
Highly recommended
reading
PHILIP TETLOCK STUDYING 15,000
GEO-POLITICAL PREDICTIONS

51. 0.06
0.05
0.04
0.03
0.02
0.01
0
Experts Dillettantes
(non
experts)
Chimps
(random
guesses)
Descrimination
Philip Tetlock
EXPERT POLITICAL JUDGEMENT

52. HOW MANY INVESTMENT GURUS STOCK
MARKET PREDICTIONS WERE CORRECT?
SCORE BELOW 50% = 1 POINT
48%
2% less accurate than a coin toss!

53. HOW TO WORK THE CROWD!
EXPLORING THE BEST TECHNIQUES TO USE TO EXTRACT RELIABLE PREDICTIONS

54. MAKE IT REWARDING
350%
300%
250%
200%
150%
100%
50%
0%
-50%
-100%
-150%
-200%
Evaluation of 20 different ads
Monadic rating
0.89 correlation 5x differentiation

55. WEIGHT BASED ON CONFIDENCE
33%
Prediction accuracy
37% 36%
44%
50%
40%
30%
20%
10%
0%
Total Guess Have a Hunch Fairly Sure Very Confident

56. MONEY BET IS A PROXY
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Bet amount: correlation with outcome
-3 -2 -1 +1 +2 +3

57. USE PREDICTION MARKETS
IOWA ELECTRONIC MARKET 480/590 OUT PREDICTED THE BEST POLL
SAMPLES OF UNDER 20

58. 32 HEAD TO HEAD EXPERIMENTS
A survey based approach with random cells* of 15
participants who were asked to predict which products would
sell more
vs.
15 people prediction markets trading – asked to buy or sell
variable amounts to create a confidence weighted market
* Using Montecarlo simulation technique we aggregated the predictions of 10,000 randomly
selected group of 15 participants from a larger sample to make this

59. HEAD TO HEAD COMPARISON
37%
55%
65%
70%
60%
50%
40%
30%
20%
10%
0%
Random guess Micro survey
(sample of 15)
15 people prediction
market trading
Source: GMI/Prediki based on 32 direct head to head comparisons

60. SOME ISSUES THOUGH

61. OPINIONS IN PREDICTION MARKET TRADING CAN QUICKLY
BE SET IN STONE IF NO NEW INFORMATION ISADDED
ADDING MORE PEOPLE
AFTER A CERTAIN POINT
DOES NOT CHANGE THE
RESULT

62. HEAD TO HEAD COMPARISON
37%
55%
65%
69%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Random guess Micro survey
sample of 15
15 people
prediction
markets trading
Standard survey:
sample of 200

63. INFORMATION SHARING IS KEY
PREDICTION MARKETS FEED OF INFORMATION

64. MARKETS WOULD REACT WHEN WE ADDED INFORMATION
Self generating
clues

65. THINK OF A QUESTION AS A
CONUNDRUM: INFORMATION
PROVIDES CLUES TO HELP PEOPLE
SOLVE THE PROBLEM

66. DIALECTICAL BOOT STRAPPING
ENCOURAGING CROWDS TO SELF-GENERATE THE INSIGHTS
NEEDED TO SOLVE PREDICTION CONUNDRUMS
Example = Board room decisions
Useful reference: Herzog and Hertwig (2009)

67. THE VALUE OF ADDING INFORMATION TO PREDICTIVE MARKET
37%
55%
65%
69%
81%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Random guess Micro survey
(sample of 15)
15 people Micro
prediction market
trading
Standard survey
(sample of 200)
Micro prediction
market with shared
information & free
comments
TRADING SYSTEM

68. EFFECTIVE USE OF PREDICTION MARKETS
• Incentivise - ideally with real money!
• Allow active & dynamic trading
• 16 is a crowd
• Share as much information as possible
• A moderator is important to stimulate debate and share
information
• Divide the herd: run multiple micro markets

69. SAMPLING 100’S SMALLER SMARTER GROUPS
BEING ASKED SMARTER QUESTIONS &
SHARING THOUGHTS & OPINIONS

70. SO WHO IS THE BEST PREDICTOR IN THE
AUDIENCE?
PRIZE: A REAL CRYSTAL BALL