Navigating the SEO of Tomorrow, Competitive Benchmarking, China as an e-Comme...
Jon puleston main stage - 2013
1. The Festival of NewMR 2013 would not be possible without our sponsors. Thanks to:
Gold Sponsors
Silver Sponsors
Session Sponsors
Media Partner
Festival Supporters
• Q – Research Software
• CreaMetrix Research
• TapestryWorks
• Keen as Mustard
• CrowdLab
• L&E Research
• Schlesinger Associates
• Focus Forums
• eCGlobal Solutions
• Revelation
2. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
REINVENTING THE 5 STAR
FILM RATING PROCESS
Jon Puleston VP Innovation GMI Interactive
****
”It was great”
****
“Totally Awesome”
****
”Brilliant”
****
”Amazing!!!!!”
3. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
THIS RESEARCH IS A WORK IN PROGRESS
• I am exploring this topic
• As such I would like you to view this from a
critical standpoint
• I am keen to know what you think
• Open to any ideas to help me move forward
4. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
MY AMBITION IS TO DESIGN A
BETTER FILM RATING SYSTEM
WHY?
5. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
6. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
NEARLY EVERY FILM SEEMS
TO SCORE 4 STARS!
7. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
5% 5%
30%
40%
20%
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5
DISTRIBUTION OF IMDA FILM REVIEW SCORES
WE EFFECTIVELY
DON’T USE
8. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
2nd REASON:
THIS ISSUE IS REPRESENTATIVE OF THE
WIDER ISSUE WE FACE WITH ALL
RANGE SCALES USED IN SURVEYS
They tend to deliver badly differentiated data
9. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
1 2 3 4 5
THE AVERAGE* DISTRIBUTION OF
ANSWERS ON A 5 POINT LIKERT SCALE
= slightly agree
= quite like
* This chart is based on the consolidated answers to 650 questions
10. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
BRAND ASSOCIATIONS: 5 POINT SCALE RATING
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
IT LEADS TO FLAT DATA THAT LOOKS LIKE THIS…
11. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
THINKING ABOUT WHY MOST
FILMS SCORE 4 STARS?
STARTING POINT….
12. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
WHY DO MOST FILMS SCORE 4?
NON RESPONSE BIAS:
• We mostly only watch films we think we will like
• We are potentially more motivated to rate films we enjoy
13. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
RATING OF A FILM
COMPLICATED PIECE OF THINKING
WHY DO MOST FILMS SCORE 4?
14. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
DID YOU ENJOY THE FILM?
• Yes/no = 90% in under 0.5 SECOND
• Answers are instinctive
HOW WOULD YOU RATE IT?
• Giving a 5 star rating = 2-10 SECOND
• Answers not instinctive
• Complex and subjective
15. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
BECOME A SOCIAL LANGUAGE
I ENJOYED IT
IT WAS A LETDOWN FOR ME
WARNING
I ENDORSE THIS
WE ENJOY 80% OF FILMS WE SEE
WHY DO MOST FILMS SCORE 4?
16. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
ONE OTHER ISSUE…
RATING NATURALLY PROMOTES
ANALYTICAL THINKING?
• That elevates intellectual judgements over emotional
reaction for some people (particularly men!)
17. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
18. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
0 2 4 6 8 10
Starrating
Well madeEnjoyment
19. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
HOWWELLMADE
HOW MUCH ENJOYED
ROMCOMS
FOREIGN FILMS
ART HOUSE
COMEDIES
THRILLERS
DRAMAS
BLOCK BUSTERS
HORROR
B MOVIES
MUSICALS
* Note this is an exaggerated scale for illustrative purposes only
WHY ROMCOMS DON’T WIN OSCARS?
20. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
SO IN SUMMARY
• Most films we see we are likely to like them
• The complexity of the thinking process prompts a
group to ignore the subtleties of the task and just
say I enjoyed it
• Scores are a social language and “enjoyed it” =4
• Another group judge it in an analytical/ intellectual
dimension and discount their real feelings
The result = meaningless homogenised data
21. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
SO HOW COULD WE CREATE
A MORE EFFECTIVE RATING
PROTOCOL?
22. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
STARTS WITH HAVING A THINK ABOUT
WHAT CONSUMERS WANT*
• Something that is simple and easy to interpret
• Has some context i.e. 4 stars compared to what?
• Takes account of genre: e.g. a thriller/romcom/art house
• Can be trusted
• Takes account of peoples individual taste & preferences
• Provides an assessment of the risk of whether you will enjoy it
• Provides an understanding of what a film will offer you – clues as to how
we can enjoy a film
• Pragmatic but not patronising
* Based on 20 qualitative in depth interviews with various film lovers including my
Mum, girlfriend, daughter and the 3 people I sit next to in the office! (i.e. I am not
saying this this is wholly comprehensive and objective feedback!)
23. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
LOOKING AT ALTERNATIVE APPROACHES FROM A
MARKET RESEARCH PERSPECTIVE WHAT WE HAVE
BEEN EXPLORING ARE 5 AVENUES:
1. Using more descriptive scales
2. More creative questioning approaches
3. Alternative judgement dimensions
4. Exploring comparative evaluation techniques
5. A Linguistic scoring system
24. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
USING MORE DESCRIPTIVE SCALES
25. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
LEADS TO WONKY ANSWER DISTRIBUTION
FOUND NO SATISFYING SOLUTION
26. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
GREAT
QUITE
GOODBRILLIANT
TV
MOVIE INSPIRING
GOT LOST IN CHOICES – MESSY DATA
BORING FUNNY
EXPLORED A MULTI CHOICE EVALUTION PROCESS
27. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
EXPERIMENTED WITH A RANGE OF
MORE CREATIVE SCORING TECHNIQUES
28. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
GOOD FOR REDUCING PATTERN ANSWERING IN
REPETITIVE QUESTIONS
0
0.05
0.1
0.15
0.2
0.25
0.3
1 2 3 4 5 6 7 8 9 10
Balance of data
Heart Grid
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5 6 7
Grid format Interactive format
Patter answers
BUT THAT IS NOT REALLY THE PROBLEM TRYING TO SOLVE IN THIS CASE
29. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
EXPLORED ALTERNATIVE MEASUREMENT DIMENSIONS
JUST AS FLAT DATA: 70% OF FILMS MET PEOPLES EXPECTATIONS
30. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
ALTERNATIVE MEASUREMENT DIMENSIONS
DIFFICULT TO DETERMINE WHAT THIS MEANT
31. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
CHOICE BASED ASSESSMENT
32. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
CHOICE BASED ASSESSEMENT
33. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
TRADITIONAL MONADIC STAR RATING
34. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
Pleas note: this can only be treated at anecdotal evidence based on small pilot
CHOICE BASED
DIFFERENTIATING BUT COMPLEX
35. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
SELF ANCHORED CHOICE COMPARISONS
WAS GRAVITY BETTER OR WORSE THAN SPACE ODYSSEY
NICE BUT STILL QUITE COMPLICATED!
36. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
GATHERED 10,000 FILM REVIEWS ACROSS 8 ENGLISH SPEAKING COUNTRIES
LINGUISTIC ANALYSIS
37. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
ANALYSED THE CHARACTER OF FEEDBACK*
COMPLAINTS
DAMNING REMARKS
BASIC PRAISE
SUPERLATIVES
ACCOLADES
e.g. Fantastic
e.g. The best
HOW IT MADE ME FEEL
HOW IT MADE ME THINK
OBSERVATIONS
NONCHALANCE
e.g. I enjoyed/Exciting
Scary/Funny/Sad etc
e.g. Thought provoking
e.g. It was good
e.g. Well Directed
e.g. Not bad
e.g. Slow
e.g. Rubbish
Highest scoring
Lowest scoring
* Based on the analysis of 10,000 short films reviews
38. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
Enthusiometer
For market researchers of quality & distinction
39. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
DEVELOPING A WORD SCORING SYSTEM
a system based upon respondents’ ratings of words
super
great
gem
thrilling
fascinating
inspiring
exciting
very good
beautiful
clever
entertaining
engrossing
enjoyable
enchanting
innovative
classic
amusing
hilarious
moving
interesting
adventurous
touching
good
fun
cool
incredible
pretty good
heart-warming
very nice
emotional
liked
provoking
quite good
decent
quirky
different
fine
nice
bizarre
average
typical
ordinary
confusing
light weight
childish
juvenile
dated
overrated
dull
not great
nothing
poor
disliked
boring
mindless
disappointing
stupid
exceptional
masterpiece
spectacular
fantastic
outstanding
superb
excellent
amazing
greatest
brilliant
awesome
wonderful
fabulous
epic
powerful
favourite
loved
marvellous
best
absurd
bad
very poor
nonsense
pointless
useless
lousy
sucked
drivel
pathetic
dreadful
garbage
junk
awful
rubbish
terrible
crap
the worst
disgrace
4 points
3 points
3 points
2 points
2 points
1 points
-2.5 points
-2.5 points
-0.5 points
-3 points
“extremely ****”
“really ****”
“quite ****”
“incredibly ****”
multiply by 1.75
multiply by 1.5
multiply by 0.5
multiply by 2
“not ****”
multiply by -0.5
40. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 3 5 7 9 1113151719212325272931333537394143454749515355
5 star scorring system language scoring system
This is how the natural language scoring
system correlates with actual ratings…
41. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
AUGMENTED WITH A NET PROMOTER STYLE APPROACH
SUPERLATIVES COMPLIMENTS MEDOCRITY NEGATIVES
amazing
awesome
brilliant
classic
excellent
powerful
great
interesting
love
beautiful
satisfy
good
like
nice
okay
fine
all right
annoy
bad
boring
crap
overrated
rubbish
% of
POSITIVE
WORDS
subtract
% of
NEGATIVE
WORDS
42. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
LINGUISTIC NET PROMOTER DELIVERED FAR
GREATER DIFFERENTIATION OF SENTIMENT
43. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
“good sci-fi story with excellent special effects”
WHAT STOOD
OUT ABOUT IT
WHAT TYPE OF
FILM IT WAS
“it was hilarious. i loved the minions and couldn't
stop smiling”
“great film, most enjoyable whilst playing
my emotions”
“some excellent action, a reasonable plot
although a little transparent”
AN OPEN-ENDED REVEALS MORE THAN
JUST SENTIMENT & RATINGS
44. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
INFORMATION WE COULD MINE FROM THE COMMENTS
CEREBRAL IMPACT e.g. challenging
AESTETICS e.g. Cool, Beautiful
ACTING/DIRECTING/STORY e.g. great acting
ETHUSIASM e.g. Fantastic
LOVE/HATE e.g. Loved it/it was OK
MEETING EXPECTATION e.g. Not as bad as a I thought
CHAT VALUE e.g. Number of words written
CONNECTION LEVEL e.g. I enjoyed v it was enjoyable
EMOTIONAL IMPACT e.g. heart warming
HUMOUR e.g. Hilarious
PERSONALITIES e.g. Love De Niro
45. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
BEGAN TO CODE & CLUSTER COMMENTS BY THEMES
HUMOUR
“funny”
“hilarious”
“humorous”
“made me
laugh”
“comical”
“light-
hearted”
ACTION
“action”
“action-
packed”
“lively”
“explosive”
“exciting”
“high-
octane”
DRAMA
“emotional”
“romantic”
“heart-
warming”
“touching”
“inspiring”
“love”
“dramatic”
TENSION
“tense”
“thrilling”
“jump”
“suspenseful”
“scary”
“dark”
“intense”
“anxious”
THEMATIC DIFFERENCES
46. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
EMOTIVE COGNATIVEVS
“emotional”
“heart-
warming”
“touching”
“happy”
“love”
“joy”
“interesting”
“educational”
“thought-
provoking”
“engaging”
“moralising”
“values”
BEGAN TO CODE & CLUSTER COMMENTS BY THEMES
47. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
Emotion
93%
Reason
7%
Emotion
29%
Reason
71%
Twilight Breaking Dawn – Part 2 The Hunger Games
BECOMES A DISCRIPTIVE TOOL
48. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
A DESCRIPTIVE TOOL
see where films fits in
120%
MORE
STORY*
148%
MORE
ACTION*
150%
MORE
ACTING
SKILL*
63%
MORE
DIRECTION
& STYLE*
174%
MORE
STORY*
* COMPARED TO A TYPICAL ACTION MOVIE
49. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
A DESCRIPTIVE TOOL
see where films fits in
81%
MORE
DRAMA
*
82%
MORE
HUMOUR*
159%
MORE
TENSION*
200%
MORE
VISUAL
APPEAL*
118%
MORE
TENSION
* COMPARED TO A TYPICAL COMEDY * COMPARED TO A TYPICAL DRAMA
50. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
SUMMARY OF WHAT WE HAVE LEARNT
• Realise that we are not quite there yet
• What we realised is that the challenge is not to
substitute this measure but enhance it and make it
more meaningful through better contextualisation
• What is actually needed is a more multi dimensional
measure
• This needs to be simple and universally easy to
understand
51. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
NEXT STEPS
• Rolling out thinking to different categories & topics
• Doing some experiments in different languages
• Working on developing some Infographics to
communicate a more multidimensional measure
» We then plan to test these out on cinema goers to see
what they find most useful
• Preparing a proper paper on all the learnings
52. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
Q & A
Jon Puleston
GMI Interactive
Lenny Murphy
GreenBook
53. Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
Jon Puleston
Email: jpuleston@gmi-mr.com
TWITTER @JONPULESTON
www.question-science.blogspot.co.uk