Will goodhand advanced quant - 2011 - 1

Event
sponsored
by
Affinnova

All
copyright
owned
by
The
Future
Place
and
the
presenters
of
the
material

For
more
informa=on
about
Affinnova
visit
h>p://www.
affinnova.com/

For
more
informa=on
about
NewMR
events
visit
newmr.org

Advanced
Quant
Techniques

July
14,
2011

9,000
Concepts
Later:

How
the
wisdom
of
the
crowd
compares

with
monadic
screening

Will
Goodhand,
Brainjuicer

2
9,000 Concepts Later:
How the wisdom of the crowd
compares with monadic
screening

Will Goodhand, BrainJuicer, UK
NewMR Advanced Quant Techniques, July 14, 2011Feb 2011
3
S  IEM is 500 non-rep
Americans betting on
election outcome
S  IEM predictions of
600+ elections over 20
years were more
accurate than most
accurate opinion poll
75% of the time
S  IEM shows Predictive
Markets potential vs.
MR s current gold
standard (large, rep
samples etc)
S  Francis Galton
1822-1911 (self-
professed elitist)
unexpectedly
discovered crowds
could be wise, whilst
trying to prove that
democracy was a poor
idea
S  Guess the weight of the
Ox showed a crowd of
800 was essentially
perfect [1197 vs.
1198lb], better than any
individual (including
livestock experts)
within it
The Wisdom of Crowds
The Wisdom of Crowds
The crowd is wise when, Diverse,
Independent & faithfully aggregated
James Surowiecki (2004)

4
Market Research Application
Predictive Markets
NewMR Advanced Quant Techniques, July 14, 2011

5
Predictive Markets:
Changing the rules of concept screening; 3 main benefits
Convenience sample of 500,
not targeted
Ask how market will react,
not personal preference
Comparative x15, not monadic
Proven Accuracy
Greater discrimination
Spots breakthrough ideas

6
Predictive Markets – ‘Imagine you owned shares in all these ideas...’
Probably Sell Shares In... Probably Buy Shares In...
Which One To Sell All Shares In... Which One To Double Shares In...
Net Preference% %
Full evaluation of
‘reasons why’
Full evaluation of
‘reasons why’
Provides
Normable
Benchmark

7
-5 -4 -4
-2
-4
-2 -2
-4 -4
-2
-7
-5 -5
-9 -9 -9
-14
-20
-5
-9
-7
-12 -12
-20
16
12 12
9
7 7
5
7
5
2
5
2 2
4 4 4 4 4
11 12
9
11
9
14
11
9 9
7
4
-5
-4
-2
2
4
5
-16-11-5-5-5
-4
2 -2
-40
5
4
4
-20
-10
0
10
20
ConceptA
ConceptB
ConceptC
ConceptH
ConceptG
ConceptL
ConceptF
ConceptJ
ConceptI
ConceptE
ConceptD
ConceptT
ConceptK
ConceptO
ConceptS
ConceptN
ConceptM
ConceptW
ConceptU
ConceptR
ConceptQ
ConceptP
ConceptV
ConceptX
%ofrespondents
Net Preference
Most Successful - would double shares
Least Successful - would sell shares
And Discrimination via Net Preference = Better Decision Making
Please select which one of these ideas you would immediately sell / double your shares in
Total Sample: 889
Strength of Portfolio*
55 51
Food Portfolio Norm This Test

8
1. Proven accuracy
2. Greater discrimination
3. Spots breakthrough ideas
Predictive
Markets

9
In market validation – 4 Quick Case Studies
Green lighted ideas that have gone on to perform well in market….

10
Picks out the Same Winners as Monadic Testing
Monadic Ideas Tested No. of Green Lighted Concepts
Snacks 15 5 out of 5
Household cleaners 10 5 out of 5
Air Care Products 5 2 out of 2
Non-alcoholic Drinks 8 1.5 out of 2#
Still Waters 11 1 out of 2#
Average Correlation 0.91
Sequential No. of Green Lighted Concepts
Telecoms 42 8 out of 11
Chilled Food 11 3 out of 3
Cleaners 29 6.5 out of 9
Chewing gum 7 4 out of 4
Lighting ideas 20 9 out of 10
Air Care Products 28 5 out of 7#
Average Correlation 0.81
#
0.5
indicates
a
concept
given
a
green
light
in
tradiGonal
test
and
pause
for
thought
in
PM
(i.e.
progress
aKer
re-‐framing)

11
0
10
20
30
40
50
60
70
80
90
100
%ofrespondents
Concept A on 97th percentile
vs Predictive Market Probably
Buy Norms
Concept A on 96th percentile
vs Monadic Concept Test
Purchase Intention Norm
Predictive Market
Normative Database
Monadic Concept Test
Normative Database
97th percentile 96th percentile
PM consistent with later stage traditional monadic testing

12
Most
encouraging
Least
encouraging
Encourage to find out more
Least
likely
Most
likely
Likelihoodtostopandread
PCE
BM
PPF
SW
NR
MH
Predictive Even in Niche Markets:
Crowd Selected Same Top 3 as Rheumatologists
Net Preference Score
SW
MHBM
Monadic Test Key Metrics
Highly Targeted Audience = Rheumatologists
Predictive Market Key Metrics
Convenience Sample of 500
= Same Result
%Happiness
NR
PCE
PPF
Spearman s Rank Correlation
Net Pref vs Rheumatologists
survey overall metric = 0.89
Represents a 90% reduction in the cost of medical concept testing

13
Comparability of tests for screener over time
Yes; probably buy / probably sell sifting question allows comparisons over time, in
different contexts, and is normable
35 38 39 39 41 45 46 48 49 50 51 52 57 61 62
48
Bagged
Snack
39 40 45 46 47 47 48 51 54 54 62
47
Bagged
Snack
36 40 41 41 41 42 44 48 49 50 5348
Bagged
Snacks
%
probably
buy

Bagged
Snack
Bagged
Snack
Bagged
Snack
Test 1
Test 2
Test 3

More Stable Cross Country Norms
53Germany59Germany
54UK52UK
56US56US
59France70France
17Range48Range
50Min43Min
67Max91Max
58Russia68Russia
67Turkey65Turkey
50Netherlands43Netherlands
67Central/South America91Central/South America
59Spain60Spain
Food &
Bevs %
probably
buy more
shares
Country
Food &
Bevs %
top2box
PI
Country
PREDICTIVE MARKETS
NORMS
CONCEPT OPTIMIZER
NORMS
S  Allows easier cross country comparisons and means fewer databases have to be kept

CONCEPT OPTIMIZER
NORMS
PREDICTIVE MARKETS
NORMS
Category
All UK
concepts
% Top2Box
PI
Category
All UK
concepts
%
probably
buy more
shares
Average
Average
Food 54 Food 54
Technology 60 Technology 60
Household 61 Household 52
Beverages 47 Beverages 55
Toiletries 55 Toiletries 55
Games/Toys 42 Games/Toys 58
Max 61 Max 60
Min 42 Min 52
Range 19 Range 8
More Stable Cross Category Norms too

16
1. Proven accuracy
Predictive
Markets

17
Greater discrimination vs. monadic tests
Total Sample: 500
-9 -7
-10
-3
-30
31
19 17
7 7
13
6
22
12
7
4
-26
-14
-24
-13
-7
-35
-25
-15
-5
5
15
25
35
Concept 1 Concept 2 Concept 3 Concept 4 Concept 5 Concept 6 Concept 7
%ofrespondents
0
10
20
30
40
50
60
70
80
90
100
SequentialMonadic%Def/Prob
Net Preference
Flat line monadic scores
Wheat from chaff

18
Better Distinguishes Wheat from Chaff
(normalised range for each test type)
Scores shown represent normalised range ie. the range divided by the standard error
(of the % net preference in the PM or %TTB in the monadic test)
Monadic Ideas
Tested
Range of Net Preference
[Predictive Markets]
Range of PI
[Monadics]
Air Fresheners 5 19.6 4.8
Savoury snacks 15 26.0 6.1
Household cleaners 10 26.9 10.7
Average 24.2 7.2
Sequential
Telecoms UK 14 12.8 7.8
Telecoms France 14 14.2 8.9
Telecoms Spain 14 20.3 18.4
Chilled Food 11 14.3 14.0
Window cleaners 17 27.6 18.1
Chewing gum 7 20.8 4.1
Cereals 11 17.2 2.0
Average 18.2 7.3

19
-‐24

-5
-7 -7 -6
-10 -11
-9
-17
-28
20 20
16
13
8 8
4
7
4
15 13
9
7
-5
-9
-4-3
-25
-20
-15
-10
-5
0
5
10
15
20
25
Concept 1 Concept 2 Concept 3 Concept 4 Concept 5 Concept 6 Concept 7 Concept 8 Concept 9
%ofrespondents
Net Preference
Nigeria:
Very clear results even in a high scoring ‘everyone is a winner’ culture
Strength of Portfolio
69 76
Portfolio Norm (Sub
Saharan Africa)
This Test
Performance
v
norms

Top
Quar5le

2nd
Quar5le

Bo9om
50%

%s
based
on
total
seeing
each
concept:
c500

20
1. Proven accuracy
Predictive
Markets

21
Amber Lights: Alvin and the Chipmunks
S  The
Predic=ve
Market
revealed
this
as
a

polarising
film
idea,
with
detractors

seeing
it
as
boring
and
only
for
nostalgic

baby
boomers
and
proponents
seeing
it

as
being
great
for
children
and
families.

S  The
concept
clearly
had
energy
with
a

large
propor=on
of
the
sample
choosing

to
trade
the
idea.
17%
traded
Alvin
&

the
Chipmunks
vs
an
expecta=on
of
less

than
12%.
A
classic
light
concept
with

poten=al
given
the
right
targe=ng

−  “Sorry,
it
is
about
a
boring
subject,
can't

be
improved.”

−  “Done
right,
this
flick
has
enormous

poten5al.”

−  The
predic=ve
market
indicated

−  targe=ng
would
be
key;
families,

baby
boomers
with
grandchildren

−  music
should
appeal
to
this
target

group

−  ‘outdated’
Chipmunks
need
to
be

completely
refreshed
for
this
film
to

work

-2 -2
-4 -3
-5
-8
-4 -5 -4 -4 -3 -3
-9
-11 -10 -11
-13
19
16
12
9
6
9
5 5 4 3 2 1
3 2 1 1 2
-25
-20
-15
-10
-5
0
5
10
15
20
25
I Am
Legend
National
Treasure
The Golden
Compass
Charlie
Wilson's
War
P.S. I Love
You
Alvin and
the
Chipmunks
Sw eeney
Todd
The
Perfect
Holiday
The Bucket
List
The Water
Horse
The Great
Debaters
Atonement Aliens Vs.
Predator
The Kite
Runner
Youth
Without
Youth
JUNO Walk Hard
%ofrespondents
Net Preference

22
Impressive Record of Predicting In-Market Reality
BoxOfficetakings
Correlation 0.81
Net Preference
$103.3
$78.4
$31.9
$22.7
$14.3
$55.9
$18.7
$3.2
$0.8
$29.6
$0.7 $0.1 $0.6
$8.8
$14.7
$1.1
$19.1
-15
-5
5
15
25
35
45
55
65
75
85
95
105
115
I Am
Legend
National
Treasure
The Golden
Compass
Charlie
Wilson's
War
P.S. I Love
You
Alvin and
the
Chipmunks
Sw eeney
Todd
The
Perfect
Holiday
The Bucket
List
The Water
Horse
The Great
Debaters
AtonementAliens Vs.
Predator
The Kite
Runner
Youth
Without
Youth
JUNO Walk Hard
TicketSales(inmillions)
17 14 9 6 1 1 1 0 -1 -1 -1 -1 -5 -9 -9 -10 -11
S  PM predicted 2 top films of 17 (in right
order) & 3 of top four
S  Alvin & the Chipmunks a classic
( pause for thought ) polarising
concept = potentially breakthrough
idea, which turned out to be successful.
0
20
40
60
80
100
120
-30 -20 -10 0 10 20 30
Industry Benchmark for Success

23
Confirming what we know about Predictive Markets
Predictive Markets
Proven accuracy
Greater discrimination
Stable norms
Can spot breakthrough ideas
Rich qualitative diagnostics
Cost effective 15 ideas x 500
2 weeks from brief to debrief
Monadic Testing
═  Equally accurate for top ideas
˜  Less discrimination
˜  Variable to volatile norms
˜  Likely to miss breakthrough ideas
˜  Thin open-end diagnostics
˜  Expensive 1 idea x 150
˜  5 weeks from brief to debrief
Comparing
400+ Concepts
20+ Categories
Award
Winner
2007
2005
—  Less variability cross countries
—  Less variability cross categories
—  Little subgroup variation
—  Effective in highly specialized categories
—  Wide range of potential applications

Q & A
Will Goodhand
BrainJuicer
Sue York
The Future Place

25
Contact us
www.brainjuicer.com
Will Goodhand
Juicy Evangelist
will.goodhand@brainjuicer.com
+44 7980 673 186

Will goodhand advanced quant - 2011 - 1

Recommended

Recommended

More Related Content

Similar to Will goodhand advanced quant - 2011 - 1

Similar to Will goodhand advanced quant - 2011 - 1 (20)

More from Ray Poynter

More from Ray Poynter (20)

Recently uploaded

Recently uploaded (20)

Will goodhand advanced quant - 2011 - 1