This document discusses new techniques for mining respondents for information through survey design. It advocates treating surveys more like entertainment to engage respondents. Examples are provided of reframing questions to encourage richer, more thoughtful feedback through things like adding context, time limits, games, and visuals. Implicit measures are discussed as a way to understand unconscious brand associations. Natural language processing is opening opportunities to analyze open-ended responses at scale. Predictive techniques like asking people to predict outcomes can provide more accurate insights than direct questions about intentions. The document promotes an approach of engaging and challenging respondents through creative, game-like survey design.

1. MINING THE BRAIN FOR INFORMATION:
NEW RESEARCH TECHNIQUES
By JON PULESTON
VP Innovation

2. HOW TO BETTER MINE RESPONDENTS
FOR INFORMATION STARTS WITH
LEARNING HOW TO EFFECTIVELY
COMMUNICATE WITH THEM

3. THINK ABOUT SURVEYS LIKE
ADVERTISING AGENCIES THINK ABOUT
ADVERTISING

4. NEED TO THINK OF SURVEYS AS A
PIECES OF ENTERTAINMENT

5. WHICH SURVEY DO YOU WANT TO COMPLETE?

6. OR THIS ONE?

7. RESPONDENTS ARE CONSUMERS

8. MOTIVATING THEM TO THINK
HOW WE RAISE PEOPLE’S INTEREST AND GET THEM THINKING

9. HERE IS HOW A TYPICAL SURVEY STARTS...

10. WE JUMP STRAIGHT INTO THE NITTY-GRITTY

11. WHAT’S YOUR SURVEY’S CHAT-UP LINE?
“Which of the following
confectionary products have you
purchased for yourself or a family
member in the last 2 weeks
darling?”

12. IF YOU HAD TO DO SOME RESEARCH ON
TOOTHPATE, HOW WOULD YOU START THE
SURVEY?
Which brands of toothpaste are
you aware of?

13. …OR LIKE THIS?
Which of these celebrities has
got the nicest teeth?

14. COPYWRITING QUESTIONS

17. RESPONDENT REACTION?

18. HOW YOU ASK A QUESTION
CAN HAVE A BIG IMPACT
Please describe your favourite meal
Imagine if you were on death row and
had to plan your last meal. What would
you order?

19. THE RESULT: RICHER MORE THOUGHTFUL FEEDBACK
A question
A game
Steak au pouivre
Scotch broth soup as a starter served with garlic bread.
Medium grilled gammon steak with a lightly fried egg on
top with chips and side salad. A glass of red wine. A sticky
toffee pudding, followed by cheese and biscuits.
Pesto pasta
Fish and chips
Garlic chicken
Bacon chips and tinned tomatoes and an egg butternut
squash soup(homemade) fillet steak well done (not burned)
new potatoes fresh garden peas fresh fruit salad and cream
all served with a good white wine
Calamari for starters Curried goat with rice & jerk chicken
with plantain and fried dumplings for main meal and hot
sticky toffee treacle pudding with hot custard for dessert
with a triple amaretto and lemonade to drink
Classic roast dinner with beef cooked medium rare, pink in
the middle and a little bit of blood, the roast potatoes crispy
on the outside soft on the inside, Yorkshire puddings, peas
and gravy cooked in shallow oil and chips

20. LIKERT QUESTION EXAMPLE…
How much do you agree or disagree
with these statements?
How much like you are these people…

22. THE APPLICATION OF RULES
Abstract rules are the core constituent of most games

23. STRIP AWAY RULES FROM GAMES
AND WHAT DO YOU HAVE?
Golf
A long walk with a heavy bag
Football
Run around for hour and a half

24. Example...
Describe yourself?
Describe yourself in EXACTLY 7 words
2.4 85%
4.5 98%
Descriptors
Descriptors
Answered
Answered

25. UNDERSTANDING THE VALUE OF RULES IN
QUESTION
In LESS THAN 7 words
Name ONE THING you would change about this product
Pick EXACTLY 3….
If you were buying xxxx which of these things would you DEFINITELY
do prior to purchase
If you could only buy one of these brands for the REST OF YOUR LIFE
In your PERFECT world what would you do…

26. ART DIRECTING

27. GETTING TO GRIPS WITH THE EFFECTIVE USE
OF IMAGERY IN SURVEYS

28. THIS SORT OF APPROACH IS NOT GOOD ENOUGH

29. IMAGES CAN ADD MEASURABLE VALUE
WHEN USED PROPERLY...

30. WHAT THEY CAN DO IS ENCOURAGE
RESPONDENTS TO STOP AND THINK!

31. Resulting in richer feedback
35% more
drinking incidents
recorded across the
day

35. Pain sliders

36. Pain sliders

37. Pain sliders
Source: ESOMAR Asia April 2012
Can gamification techniques cross cultures

38. EXAMPLE OF THE ERGONOMIC DESIGN OF
QUESTIONS

39. UNDERSTANDING THE ROLE OF ICONOGRAPHY
LESS LOAD ON THE WORKING MEMORY

40. MAKING TASKS MORE INTELLECTUALLY SATISFYING
Issues with this question:
1. Boring and repetitive so people’s brains switch off

41. India
Commonwealth Games
Cricket World Cup
Football World Cup
Indian Premier League (ILP)
Wimbledon Tennis Championships
Formula One (F1) Motor Racing
Oscars Ceremony/Academy Awards
Olympic Games (Summer)
English Premier League (Football)
Olympic Winter Games
Rugby World Cup
World Athletics Championships
MTV Video Music Awards
UEFA Champions League
Youth Olympic Games
Barclays Premier League
NBA Championships
The Masters (Golf)
Paralympic Games
NFL Superbowl
Extreme Sports X Games
NHL Stanley Cup
UK
3.3%
3.3%
5.4%
5.4%
5.4%
6.5%
6.5%
12.0%
13.0%
15.2%
17.4%
21.7%
23.9%
29.4%
30.4%
32.6%
33.7%
35.9%
38.0%
52.2%
55.4%
66.3%
16.2%
26.7%
13.3%
62.9%
9.5%
11.4%
18.1%
12.4%
13.3%
13.3%
21.0%
21.9%
33.3%
15.2%
52.4%
20.0%
30.5%
19.1%
11.4%
33.3%
56.2%
70.5%
12% of people in the UK claim be unaware of the Olympics?

42. INTELLECTUAL SATISFACTION
A MORE PURPOSEFUL & REWARDING QUESTION
TO ANSWER

43. UK
Football World Cup
Olympic Games (Summer)
Formula One (F1) Motor Racing
MTV Video Music Awards
Oscars Ceremony/Academy Awards
UEFA Champions League
Barclays Premier League
Commonwealth Games
Olympic Winter Games
Paralympic Games
Rugby World Cup
The Masters (Golf)
Wimbledon Tennis Championships
World Athletics Championships
Cricket World Cup
English Premier League (Football)
NBA Championships
Extreme Sports X Games
NFL Superbowl
Indian Premier League (ILP)
Youth Olympic Games
NHL Stanley Cup
India
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.9%
0.9%
0.9%
0.9%
0.9%
0.9%
0.9%
0.9%
1.7%
1.7%
2.6%
3.5%
3.5%
7.0%
7.8%
12.2%
0.0%
3.3%
3.3%
5.4%
1.1%
8.7%
9.8%
1.1%
5.4%
6.5%
3.3%
4.4%
3.3%
3.3%
0.0%
5.4%
10.9%
9.8%
14.1%
1.1%
2.2%
16.3%

44. MAKING QUESTIONS MORE
REWARDING

45. RESPONDENT HAD TO PREDICT THE BRAND AGAINST
THE CLOCK & WIN POINTS FOR GETTING IT RIGHT

46. AWARENESS TASK: ANSWER TIME PER QUESTION
Standard Survey
Guessing Game with time pressure

47. MORE ACCURATE BRAND RECOGNITION

48. HOW TO GET RESPONDENTS TO THINK
IN DIFFERENT WAYS
Harder
faster
intuitively
Objectively
creatively

49. CONCIOUS DECISIONS
UNCONCIOUS DECISIONS
MOST QUESTIONS IN SURVEYS MINE
ANALYTICAL THINKING WHICH ONLY A
FRACTION OF OUR DECISION ARE BASED
UPON

50. SURVEYS TEND TO BE DOMINATED BY
SYSTEM 2/ANALYTICAL JUDGEMENTS
Which of these pet food products
would you buy?
Research says this
The one with the
best ingredients
Sales say this
The one with the nice
picture of a dog

51. CAN BE AS SIMPLE AS IMPOSING TIME
LIMITS TO CHANGE THINKING PROCESS

52. BRAND CHOICE

53. TECHNOLOGY IS DRIVING THE RAPID
GROWTH OF IMPLICIT MEASUREMENT
TECHNQUES

54. IMPLICIT ASSOCIATION TESTS
2 second
1/28TH second
4 second
Brand logo
Impressions of person subliminally influenced by brand logo.
Test against a blank image or different brand or pre and post advertising

55. Impact of brand on perceptions of person
Agree
Disagree
Very ordinary
Con artist
I don’t need in my life
Company I would enjoy
Annoying
Care about me
Someone I could count on
Someone I could relate to
Only interested in making money
Likable
Exciting
-30% -20% -10%
0%
10%
* Based on comparison of net scores from top 2 & bottom 2 choices on 7 point scale v control
20%
30%

56. MEASURING IMPLICIT EMOTIONAL
RELATIONSHIPS WITH BRANDS
Cheerful
Warm
OldReliable
Fashioned
Fun
SOURCE: COG RESEARCH

57. Agree
Cool
Cheerful
Fun
Reliable
Warm
Slow
Fast
Old fashioned
Disagree

58. Agree
Cool
Cheerful
Fun
Reliable
Warm
Slow
Fast
Old fashioned
Disagree

59. USING FACIAL RECOGNITION TECHNOLOGY WE CAN NOW
MEASURE HOW PEOPLE FEEL WITHOUT ASKING QUESTIONS

60. OPEN YOUR MIND
TO THE POWER OF
THE
OPEN
ENDED
QUESTION
WWW.GMI-MR.COM | CONFIDENTIAL
| 60

61. A BI-PRODUCT OF BIG DATA ARE RAPID
ADVANCEMENTS IN NATURAL LANGUAGE
PROCESSING TOOLS & TECHNIQUE
THIS TECHNOLOGY IS OPENING UP NEW OPPORTUNITIES
FOR MARKET RESEARCHERS

62. 50% OF FILMS SCORE 4 STARS
DISTRIBUTION OF STARS
rush
skyfall
olympus has
fallen
harry potter
now you see me
elysium
despicable me 2
kick ass 2
alpha papa
iron man 3
the dark knight
runner runner
the way way
back
freshman father
brother bear
she was to good
for him
lawless
the muppets
les miserables
jack reacher
about time
great gatsby
looper
st trinians
8 degrees below
the butler
mama
the heat
meet the
robinsons
the rush
this is the end
the world's end
transformers
red obsession
incredibles
mirror boy
red2
mortal
instruments
borat
mud
scooby doo 2
american pie
i'm so excited
narnia
battleship
next three days
the boat that
rocked
no pain no gain
meet the milners
the croods
hansel and gretel
the flying
scotsman
crossing over
the last mile
oblivion
the notebook
i give it a year
the sweeney
avatar
the woods
only the rain
think like a man
act like a lady
visitor q
tower heist
weekend
twilight
epic
behind the
candelabra
papillion
end of watch
plan 9 from outer
space
puss in boots
red state
the conjuring
we're the millers
planes
one direction this
is us
the awakening
grown ups 2
man of steel
smurfs
monsters
university
taken 2
anuvahood
hang em high
after earth
pitch perfect
the lone ranger

63. WORD SCORING SYSTEMS
a system based upon respondents ratings of words
3 points
4 points
exceptional
masterpiece
spectacular
fantastic
outstanding
superb
excellent
amazing
greatest
brilliant
awesome
wonderful
fabulous
epic
powerful
favourite
loved
marvellous
best
super
great
gem
thrilling
fascinating
inspiring
exciting
very good
beautiful
clever
entertaining
engrossing
enjoyable
enchanting
innovative
classic
amusing
hilarious
moving
3 points
2 points
2 points
interesting
adventurous
touching
good
fun
cool
incredible
pretty good
heart-warming
very nice
emotional
liked
provoking
quite good
decent
quirky
different
fine
nice
1 points
“incredibly ****”
multiply by 1.75
“really ****”
multiply by 1.5
“quite ****”
multiply by 0.5
bizarre
average
typical
ordinary
confusing
light weight
childish
juvenile
dated
overrated
dull
not great
nothing
poor
disliked
boring
mindless
disappointing
stupid
-2.5 points
multiply by 2
“extremely ****”
-0.5 points
“not ****”
multiply by -0.5
-2.5 points
absurd
bad
very poor
nonsense
pointless
useless
lousy
sucked
drivel
pathetic
dreadful
garbage
junk
awful
rubbish
terrible
crap
the worst
disgrace
-3 points

64. APPLIED TO FILMS: GREATER DIFFERENCIATION OF SENTIMENT
REFLECTING EXACTLY WHAT THE RESPONDENTS THINK

65. A SINGLE OPEN-ENDED REVEALS MORE
THAN JUST SENTIMENT & RATINGS
FILMS:
WHAT TYPE OF
FILM IT WAS
“it was hilarious. i loved the minions and couldn't
stop smiling”
“great film, most enjoyable whilst playing
my emotions”
“good sci-fi story with excellent special effects”
WHAT STOOD
OUT ABOUT IT
“some excellent action, a reasonable plot
although a little transparent”

66. CODING & CLUSTERING COMMENTS:
THEMATIC DIFFERENCES
“funny”
“hilarious”
“humorous”
“made me
laugh”
“comical”
“lighthearted”
“action”
“actionpacked”
“lively”
“explosive”
“exciting”
“highoctane”
“emotional”
“romantic”
“heartwarming”
“touching”
“inspiring”
“love”
“dramatic”
“tense”
“thrilling”
“jump”
“suspenseful”
“scary”
“dark”
“intense”
“anxious”
HUMOUR
ACTION
DRAMA
TENSION

67. ENGAGEMENT OF HEART OR MIND
“emotional”
“heartwarming”
“touching”
“happy”
“love”
“joy”
EMOTIVE
“interesting”
“educational”
“thoughtprovoking”
“engaging”
“moralising”
“values”
VS
COGNATIVE

68. OUR LANGAUGE REVEALS OUR
LESS CONCIOUS OPINIONS
Twilight Breaking Dawn – Part 2
The Hunger Games

69. 10 DIFFERENT WAYS TO DESCRIBE THE PRICE
OF A PRODUCT…
Buying purely on price
Quality trade off
Buying on quality
It’s the cheapest
Its cheap
Its cheaper than…
Its affordable
Economical
Good value
Better value
Value for money
The best value
Worth paying more for
It’s the best quality

70. PREDICTIVE
THINKING
TECHNIQUE

71. WILL YOU BUY THIS?
YES = <0.3 correlation with sales*
* Please note this is an anecdotal figure

72. WE ARE NOT GOOD AT PREDICTING OUR OWN BEHAVIOUR
WILL YOU CLEAN UP?
YES = 50%
PREDICT HOW MANY WILL CLEAN UP?
YES = 15%
HOW MANY CLEANED UP?
13%

73. THE VALUE OF THE PROJECTIVE VIEWPOINTS
How much would you pay?
£10, £11, £12, £13, £14, £15, £16, £17, £18, £19, £20+
<50% less than actual price
Predict what price you think this will be sold at?
£10, £11, £12, £13, £14, £15, £16, £17, £18, £19, £20+
Within 1% of actual price

74. ELECTION FORECASTING
WHO WILL YOU VOTE FOR?
1,000 respondents
PREDICT WHO WILL WIN?
20 people gambling

IOWA ELECTION MARKET
IN 451 out of 596 OCCASIONS

75. PREDICTION ACCURACY* = Quality of
information x Motivation of predictors to
make a correct prediction x Ability of the
people to objectively process that
information.

76. Note it’s not dependent directly on
sample size. One person with
access to all the right information
can make a correct call e.g. Nate
Silver!

77. THE MONTECARLO METHOD
The “mote carlo” method
instead of sampling lots of
people and averaging
out their answers (all of
who may have the same
network error) it samples
lots of sources of
information and averages
out their answers.

78. A MONTECARLO APPROACH TO AD TESTING:
Every measure tells a slightly different story
600
Thumbs up
500
Eye catching
400
Curiosity
Interest
300
Clarity
Positive perception
200
2 sec thumbs up
100
"This is exactly the type of advertising
we should be doing!"
Design
Winning ad
0
Considered verdict
2 sec verdict
-100

79. PREDITVE MARKET TRADING

80. GREATER DIFFERENTIATION OF OPINION
Sum Net Investment
Traditional rating
600%
500%
400%
300%
200%
100%
0%
-100%
-200%
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

81. CREATIVE THINKING

82. USING GAMES IN SURVEYS TO GET PEOPLE
THINKING MORE CREATIVELY

83. QUALITY OF IDEAS
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Very imaginative
Quite
Not very
With Stimulus
Without

84. RANDOM STIMULATION

85. 1. Idea for an idea...
6 to 18 ideas

86. 2. Thinking rooms
0.7 to 6.9 ice cream names

87. 3. Thinking through the minds of...
1 or 2 thoughts increased to 10 to 20

88. SUMMARY
• Learning how to mine our brains for
information is all about learning how to get
people think in different ways
• I would encourage you to treat respondents
in a survey like consumers
• Break away from some of the more
traditional question approaches
• Open you mind to the power of open ends
• Embrace the Monte Carlo approach
• Make things FUN!

89. Q&A
TWITTER @JONPULESTON
www.question-science.blogspot.co.uk