Jon puleston main stage - 2013

The Festival of NewMR 2013 would not be possible without our sponsors. Thanks to:
Gold Sponsors
Silver Sponsors
Session Sponsors
Media Partner
Festival Supporters
• Q – Research Software
• CreaMetrix Research
• TapestryWorks
• Keen as Mustard
• CrowdLab
• L&E Research
• Schlesinger Associates
• Focus Forums
• eCGlobal Solutions
• Revelation

Jon Puleston, GMI Interactive, UK
Festival of NewMR 2013 - Main Stage
REINVENTING THE 5 STAR
FILM RATING PROCESS
Jon Puleston VP Innovation GMI Interactive
****
”It was great”
****
“Totally Awesome”
****
”Brilliant”
****
”Amazing!!!!!”

THIS RESEARCH IS A WORK IN PROGRESS
• I am exploring this topic
• As such I would like you to view this from a
critical standpoint
• I am keen to know what you think
• Open to any ideas to help me move forward

MY AMBITION IS TO DESIGN A
BETTER FILM RATING SYSTEM
WHY?

NEARLY EVERY FILM SEEMS
TO SCORE 4 STARS!

5% 5%
30%
40%
20%
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5
DISTRIBUTION OF IMDA FILM REVIEW SCORES
WE EFFECTIVELY
DON’T USE

2nd REASON:
THIS ISSUE IS REPRESENTATIVE OF THE
WIDER ISSUE WE FACE WITH ALL
RANGE SCALES USED IN SURVEYS
They tend to deliver badly differentiated data

1 2 3 4 5
THE AVERAGE* DISTRIBUTION OF
ANSWERS ON A 5 POINT LIKERT SCALE
= slightly agree
= quite like
* This chart is based on the consolidated answers to 650 questions

BRAND ASSOCIATIONS: 5 POINT SCALE RATING
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
IT LEADS TO FLAT DATA THAT LOOKS LIKE THIS…

THINKING ABOUT WHY MOST
FILMS SCORE 4 STARS?
STARTING POINT….

WHY DO MOST FILMS SCORE 4?
NON RESPONSE BIAS:
• We mostly only watch films we think we will like
• We are potentially more motivated to rate films we enjoy

RATING OF A FILM
COMPLICATED PIECE OF THINKING

DID YOU ENJOY THE FILM?
• Yes/no = 90% in under 0.5 SECOND
• Answers are instinctive
HOW WOULD YOU RATE IT?
• Giving a 5 star rating = 2-10 SECOND
• Answers not instinctive
• Complex and subjective

BECOME A SOCIAL LANGUAGE
I ENJOYED IT
IT WAS A LETDOWN FOR ME
WARNING
I ENDORSE THIS
WE ENJOY 80% OF FILMS WE SEE

ONE OTHER ISSUE…
RATING NATURALLY PROMOTES
ANALYTICAL THINKING?
• That elevates intellectual judgements over emotional
reaction for some people (particularly men!)

0 2 4 6 8 10
Starrating
Well madeEnjoyment

HOWWELLMADE
HOW MUCH ENJOYED
ROMCOMS
FOREIGN FILMS
ART HOUSE
COMEDIES
THRILLERS
DRAMAS
BLOCK BUSTERS
HORROR
B MOVIES
MUSICALS
* Note this is an exaggerated scale for illustrative purposes only
WHY ROMCOMS DON’T WIN OSCARS?

SO IN SUMMARY
• Most films we see we are likely to like them
• The complexity of the thinking process prompts a
group to ignore the subtleties of the task and just
say I enjoyed it
• Scores are a social language and “enjoyed it” =4
• Another group judge it in an analytical/ intellectual
dimension and discount their real feelings
The result = meaningless homogenised data

SO HOW COULD WE CREATE
A MORE EFFECTIVE RATING
PROTOCOL?

STARTS WITH HAVING A THINK ABOUT
WHAT CONSUMERS WANT*
• Something that is simple and easy to interpret
• Has some context i.e. 4 stars compared to what?
• Takes account of genre: e.g. a thriller/romcom/art house
• Can be trusted
• Takes account of peoples individual taste & preferences
• Provides an assessment of the risk of whether you will enjoy it
• Provides an understanding of what a film will offer you – clues as to how
we can enjoy a film
• Pragmatic but not patronising
* Based on 20 qualitative in depth interviews with various film lovers including my
Mum, girlfriend, daughter and the 3 people I sit next to in the office! (i.e. I am not
saying this this is wholly comprehensive and objective feedback!)

LOOKING AT ALTERNATIVE APPROACHES FROM A
MARKET RESEARCH PERSPECTIVE WHAT WE HAVE
BEEN EXPLORING ARE 5 AVENUES:
1. Using more descriptive scales
2. More creative questioning approaches
3. Alternative judgement dimensions
4. Exploring comparative evaluation techniques
5. A Linguistic scoring system

USING MORE DESCRIPTIVE SCALES

LEADS TO WONKY ANSWER DISTRIBUTION
FOUND NO SATISFYING SOLUTION

GREAT
QUITE
GOODBRILLIANT
TV
MOVIE INSPIRING
GOT LOST IN CHOICES – MESSY DATA
BORING FUNNY
EXPLORED A MULTI CHOICE EVALUTION PROCESS

EXPERIMENTED WITH A RANGE OF
MORE CREATIVE SCORING TECHNIQUES

GOOD FOR REDUCING PATTERN ANSWERING IN
REPETITIVE QUESTIONS
0
0.05
0.1
0.15
0.2
0.25
0.3
1 2 3 4 5 6 7 8 9 10
Balance of data
Heart Grid
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5 6 7
Grid format Interactive format
Patter answers
BUT THAT IS NOT REALLY THE PROBLEM TRYING TO SOLVE IN THIS CASE

EXPLORED ALTERNATIVE MEASUREMENT DIMENSIONS
JUST AS FLAT DATA: 70% OF FILMS MET PEOPLES EXPECTATIONS

ALTERNATIVE MEASUREMENT DIMENSIONS
DIFFICULT TO DETERMINE WHAT THIS MEANT

CHOICE BASED ASSESSMENT

CHOICE BASED ASSESSEMENT

TRADITIONAL MONADIC STAR RATING

Pleas note: this can only be treated at anecdotal evidence based on small pilot
CHOICE BASED
DIFFERENTIATING BUT COMPLEX

SELF ANCHORED CHOICE COMPARISONS
WAS GRAVITY BETTER OR WORSE THAN SPACE ODYSSEY
NICE BUT STILL QUITE COMPLICATED!

GATHERED 10,000 FILM REVIEWS ACROSS 8 ENGLISH SPEAKING COUNTRIES
LINGUISTIC ANALYSIS

ANALYSED THE CHARACTER OF FEEDBACK*
COMPLAINTS
DAMNING REMARKS
BASIC PRAISE
SUPERLATIVES
ACCOLADES
e.g. Fantastic
e.g. The best
HOW IT MADE ME FEEL
HOW IT MADE ME THINK
OBSERVATIONS
NONCHALANCE
e.g. I enjoyed/Exciting
Scary/Funny/Sad etc
e.g. Thought provoking
e.g. It was good
e.g. Well Directed
e.g. Not bad
e.g. Slow
e.g. Rubbish
Highest scoring
Lowest scoring
* Based on the analysis of 10,000 short films reviews

Enthusiometer
For market researchers of quality & distinction

DEVELOPING A WORD SCORING SYSTEM
a system based upon respondents’ ratings of words
super
great
gem
thrilling
fascinating
inspiring
exciting
very good
beautiful
clever
entertaining
engrossing
enjoyable
enchanting
innovative
classic
amusing
hilarious
moving
interesting
adventurous
touching
good
fun
cool
incredible
pretty good
heart-warming
very nice
emotional
liked
provoking
quite good
decent
quirky
different
fine
nice
bizarre
average
typical
ordinary
confusing
light weight
childish
juvenile
dated
overrated
dull
not great
nothing
poor
disliked
boring
mindless
disappointing
stupid
exceptional
masterpiece
spectacular
fantastic
outstanding
superb
excellent
amazing
greatest
brilliant
awesome
wonderful
fabulous
epic
powerful
favourite
loved
marvellous
best
absurd
bad
very poor
nonsense
pointless
useless
lousy
sucked
drivel
pathetic
dreadful
garbage
junk
awful
rubbish
terrible
crap
the worst
disgrace
4 points
3 points
3 points
2 points
2 points
1 points
-2.5 points
-2.5 points
-0.5 points
-3 points
“extremely ****”
“really ****”
“quite ****”
“incredibly ****”
multiply by 1.75
multiply by 1.5
multiply by 0.5
multiply by 2
“not ****”
multiply by -0.5

0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 3 5 7 9 1113151719212325272931333537394143454749515355
5 star scorring system language scoring system
This is how the natural language scoring
system correlates with actual ratings…

AUGMENTED WITH A NET PROMOTER STYLE APPROACH
SUPERLATIVES COMPLIMENTS MEDOCRITY NEGATIVES
amazing
awesome
brilliant
classic
excellent
powerful
great
interesting
love
beautiful
satisfy
good
like
nice
okay
fine
all right
annoy
bad
boring
crap
overrated
rubbish
% of
POSITIVE
WORDS
subtract
% of
NEGATIVE
WORDS

LINGUISTIC NET PROMOTER DELIVERED FAR
GREATER DIFFERENTIATION OF SENTIMENT

“good sci-fi story with excellent special effects”
WHAT STOOD
OUT ABOUT IT
WHAT TYPE OF
FILM IT WAS
“it was hilarious. i loved the minions and couldn't
stop smiling”
“great film, most enjoyable whilst playing
my emotions”
“some excellent action, a reasonable plot
although a little transparent”
AN OPEN-ENDED REVEALS MORE THAN
JUST SENTIMENT & RATINGS

INFORMATION WE COULD MINE FROM THE COMMENTS
CEREBRAL IMPACT e.g. challenging
AESTETICS e.g. Cool, Beautiful
ACTING/DIRECTING/STORY e.g. great acting
ETHUSIASM e.g. Fantastic
LOVE/HATE e.g. Loved it/it was OK
MEETING EXPECTATION e.g. Not as bad as a I thought
CHAT VALUE e.g. Number of words written
CONNECTION LEVEL e.g. I enjoyed v it was enjoyable
EMOTIONAL IMPACT e.g. heart warming
HUMOUR e.g. Hilarious
PERSONALITIES e.g. Love De Niro

BEGAN TO CODE & CLUSTER COMMENTS BY THEMES
HUMOUR
“funny”
“hilarious”
“humorous”
“made me
laugh”
“comical”
“light-
hearted”
ACTION
“action”
“action-
packed”
“lively”
“explosive”
“exciting”
“high-
octane”
DRAMA
“emotional”
“romantic”
“heart-
warming”
“touching”
“inspiring”
“love”
“dramatic”
TENSION
“tense”
“thrilling”
“jump”
“suspenseful”
“scary”
“dark”
“intense”
“anxious”
THEMATIC DIFFERENCES

EMOTIVE COGNATIVEVS
“emotional”
“heart-
warming”
“touching”
“happy”
“love”
“joy”
“interesting”
“educational”
“thought-
provoking”
“engaging”
“moralising”
“values”
BEGAN TO CODE & CLUSTER COMMENTS BY THEMES

Emotion
93%
Reason
7%
Emotion
29%
Reason
71%
Twilight Breaking Dawn – Part 2 The Hunger Games
BECOMES A DISCRIPTIVE TOOL

A DESCRIPTIVE TOOL
see where films fits in
120%
MORE
STORY*
148%
MORE
ACTION*
150%
MORE
ACTING
SKILL*
63%
MORE
DIRECTION
& STYLE*
174%
MORE
STORY*
* COMPARED TO A TYPICAL ACTION MOVIE

A DESCRIPTIVE TOOL
see where films fits in
81%
MORE
DRAMA
*
82%
MORE
HUMOUR*
159%
MORE
TENSION*
200%
MORE
VISUAL
APPEAL*
118%
MORE
TENSION
* COMPARED TO A TYPICAL COMEDY * COMPARED TO A TYPICAL DRAMA

SUMMARY OF WHAT WE HAVE LEARNT
• Realise that we are not quite there yet
• What we realised is that the challenge is not to
substitute this measure but enhance it and make it
more meaningful through better contextualisation
• What is actually needed is a more multi dimensional
measure
• This needs to be simple and universally easy to
understand

NEXT STEPS
• Rolling out thinking to different categories & topics
• Doing some experiments in different languages
• Working on developing some Infographics to
communicate a more multidimensional measure
» We then plan to test these out on cinema goers to see
what they find most useful
• Preparing a proper paper on all the learnings

Q & A
Jon Puleston
GMI Interactive
Lenny Murphy
GreenBook

Jon Puleston
Email: jpuleston@gmi-mr.com
TWITTER @JONPULESTON
www.question-science.blogspot.co.uk

Jon puleston main stage - 2013

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Jon puleston main stage - 2013

Similar to Jon puleston main stage - 2013 (17)

More from Ray Poynter

More from Ray Poynter (20)

Recently uploaded

Recently uploaded (20)

Jon puleston main stage - 2013