Gamification Improves Survey Engagement and Data Quality

STRICTLY PRIVATE AND CONFIDENTIAL
TMRE2014

We have a problem…
Surveys remain the No. 1 source of data for
marketing researchers.
BUT we have less respondents willing to
take surveys or complete them.
WHY is this?
Hint:
CHASE | SOCRATIC TECHNOLOGIES | TMRE2014 2

Why it matters that surveys are boring…
It all comes down to quality and cost:
Respondent engagement is waning
 Cooperation and completion rates continue to drop
 Drop-off rates are higher
Data quality is impacted
 More respondents flat-lining
Models are less robust
 Less differentiation across attribute ratings

The Rise of Gamification…
Gamification seems to be the silver bullet,
golden ticket,
the end-all-be-all,
etc….
Many game elements do not look new –
there has been experimentation for a few years

A brief look back at historical research...
Galesic
(2006)
Drolet
(2009)
Metrixlab
(2010)
Malinoff
(2010)
Use visual
elements to
enhance the
respondent’s
experience
To keep
respondents
motivated is by
keeping the
questions
interesting
Fun and
engaging
surveys keep
respondents
engaged and willing
to participate in
future surveys
Gamified surveys
can lead to longer
completion
times
The focus and branding of gamification as a holistic set of techniques

Degrees of Gamification
Gamification can take many forms…
from restructuring traditional survey questions
to building complex game environments in which panelists can compete
The focus and branding of gamification as a holistic set of techniques

Engage!
Gamification is intended to increase engagement and better data quality
However, industry has discussed the ideas but has not implemented
We are set to change that inertia

The Test
Test the impact of engagement using an experimental design in which we test
two versions of the same survey:
Examples of ‘game-enhanced’ elements include:
 Animation
 Collage builder
 Drag & Drop

Our Hypothesis
The game-enhanced experience will result in higher levels of self-reported
satisfaction and better data quality, measured as follows:
Self-reported metrics
 Interest
 Ease of Understanding
 Fun
Data quality metrics
 Straight-lining and ‘red herring’ questions
 Comprehension
 Word count and quality of open-end responses

Design Details
Data Collection:
 n = 1,119
 Survey was fielded between August 18 and 25
Screening Requirements:
 Age 18+
 No competitive employment
 Household financial / investment decision-makers
 Household income over $25K
 Has a banking relationship

OURRESULTS

The Theory
Challenges with Testing Complex Concepts
Adding an Animation
Read a text-based
description about ETFs
Shown a short
animation about ETFs

The Findings
Game-Enhanced respondents have a higher propensity to take action
69% 68%
53% 50% 40%
Seek more information
about ETFs
At least consider
investing in ETFs
They also understand the concept better
52%
Tell a friend or family
member about ETFs
70% 69% 75% 65%
59% 51% 61% 53%
How ETFs are
different from
individual stocks
How ETFs are
different from
mutual funds
When an ETF can
be purchased on
a stock exchange
The fees
associated with
ETFs
Standard Game-Enhanced
Green text reflects a significant
difference @ 90% confidence level

The Findings
Game-Enhanced
respondents rated the
exercise more positively
This section of the
survey was interesting to me
The questions asked in this
section were easy to
understand
This section of the
survey was fun to complete
85%
70%
80%
65%
93%
71%
Green text reflects a
significant difference @
90% confidence level

OURRESULTS

The Theory
Challenges with Personal Experience Questions
Collage Building
Answer a standard
open-end question
Build a collage and
explain their selections in
follow-up open-end

The Findings
Game-Enhanced respondents had significantly higher word counts
Average Length
of Open End 16
words
23
words

Additional Insights
Game enhancement such as collage builder can solicit rich, personal
emotional experiences
“I am too scared after the 2008 downturn to invest my
money. I feel like I already have a small amount of
money to start with hence my empty pockets
selection, that I would be taking a gamble if I did
invest because what would happen? I feel like I'd
look like the guy in the picture and need to be thrown
a lifeline and the monkey's face represents how I
would feel losing money.”
“The image of the archer reminded me that although I
can have my eye on the long-term target, anything
can happen. The people balancing on the beach
remind me that investing is a balancing act between
risk and growth. The spiral staircase made me feel
dizzy and that's what it's been like trying to
understand why no one has been held criminally
responsible for the 2008 mess from the banks or
lending companies. The dice is all about gamble and
that's how I feel about investing.”

The Findings
Game-Enhanced
respondents found the
exercise more fun to
complete
understand
This section of the survey was
interesting to me
This section of the survey was
fun to complete
88%
88%
75%
77%
64%
76%
Green text reflects a
significant difference @
90% confidence level

OURRESULTS

The Theory
Common Challenge with Attribute Lists
Drag and Drop
Select from a standard
5-point scale of appeal
Drag pictures of cards
into different boxes
based on appeal

understand
This section of the
survey was interesting to me
This section of the
survey was fun to complete
86%
85%
98%
95%
89%
85%
The Findings
No differences between
respondents in the two
survey experiences

Additional insight…
Game-enhanced elements have an even bigger impact on less
frequent survey takers
A proxy for your customer lists
The panelist of the future

Moving forward!
Galesic
(2006)
Drolet
(2009)
METRIXlAB
(2010)
Malinoff
(2010)
Doran/
Yule
(2014)
Use visual
elements to
enhance the
respondent’s
experience
To keep
respondents
motivated is by
keeping the
questions
interesting
Fun and
engaging
surveys keep
respondents
engaged and
willing to
participate in
future surveys
Gamified
surveys can
lead to longer
completion
times
Game-enhanced
techniques can lead
to an increase in
data quality and
engagement but
should be used
appropriately

What you need to know
 Comprehension of complex concepts and overall data quality can be
higher when using game enhanced techniques
 Gamification exercises need to be used appropriately
 Game-enhanced questions can have a stronger impact on those who take
surveys infrequently
 Gamification techniques can re-engage respondents, help reignite interest,
and counteract survey fatigue

Gamification can help with survey length
Game-enhanced surveys can take more time. No matter how engaging the
experience, length of a survey is a major factor in respondent satisfaction.
8%
shorter than actual time
28%
shorter than actual time
Difference
in Actual
versus
Perceived
But, gamification can have an impact on the perceived length of survey.

When to use game-enhanced design
DO’s:
 Projects where the target
audience is customers/non-panelist
(‘light survey takers’)
 When testing complex concepts
and/or low interest topics
 For important open-ended
questions
DON’Ts
 Don’t use a game-enhanced
design unless there is a good
reason
 Don’t let gamification overtake the
research objectives
 Avoid using with tracking or
historical benchmarks projects

D
Cha- Cha- Cha- Changes…
Select a forward-thinking vendor
Do they have mindset for change?
Do they have the tools to set up projects ?

Gamification Improves Survey Engagement and Data Quality

Recommended

Recommended

More Related Content

Similar to Gamification Improves Survey Engagement and Data Quality

Similar to Gamification Improves Survey Engagement and Data Quality (20)

Recently uploaded

Recently uploaded (20)

Gamification Improves Survey Engagement and Data Quality

Editor's Notes