Annie Pettit - Festival of New MR - 2014

The
Fes'val
of
NewMR
2014
would
not
be
possible
without
our
sponsors.
Thanks
to:

Our
Pla'num
Sponsor
for
2014

Silver
Sponsors

Session
Sponsors

Media
Partner
Fes'val
Supporters

•  Schlesinger
Associates

•  GMI

•  krea

The
Fes'val
of
2014

Annie Pettit, Peanut Labs, Canada
Festival of NewMR, December 2014
2014
Pla9num
Sponsor

Annie
PeEt
(@LoveStats)

Chief
Research
Oﬃcer,
Peanut
Labs

VP
Research
Standards,
Research
Now

Canada

Behind
Your
Back:

What
research
par'cipants
really
think
about
researchers

Agenda

•  Are
we
being
good

ci9zens?

•  How
are
people

reac9ng
to
us?

•  Can
we
ﬁx
it?

Method

•  Social
Media
Data

–  Hypothesis
genera9on

–  N>200
000

–  Global
English

•  Survey
Data

–  Hypothesis
conﬁrma9on
and
precision

–  N=400

–  Non-‐probability
incen9vized
panel,
Sampled
and

Weighted
to
US
Census
Rep
à
Not
because
the
target

group
is
census
rep
but
because
we’re
familiar
with

census
rep

Survey
Instruc'ons

•  We
are
really
interested
in
your
honest
answers.
Don’t

worry
about
hur9ng
our
feelings
or
speaking
a
scary

truth.
You
will
not
be
penalized
for
sharing
any
of
your

opinions.
We
need
to
hear
them!

•  (Please
be
honest.
We
will
not
penalize
your
account.)

•  (Please
don't
be
scared
to
tell
the
truth.
We
will
not

penalize
your
account.)

•  (Please,
please,
tell
us
the
truth.
:)

Trust

Who
to
trust?

•  Do
ya
think
this
paid
survey
thing
is
a
scam?

•  @aaa
took
an
online
survey
once.
Seemed
legit

•  i
just
took
a
survey....
is
it
a
scam
or
is
it
just
me.

•  hey
all
pls
conﬁrm
me
is
this
trust
worthy

•  @bbb
surveys
are
scams
usually

•  @ccc
gallop
survey
totally
fraud

•  is
opinion
outpost
trust
worthy?

•  I
dont
trust
ipsos
synovate..at
all.

It’s
not
spam
if…

•  Poor
responders,
Men,
and
Older
people
are
less
likely
to
trust
us

•  Younger
people
are
more
likely
to
trust
their
friends,
and
social
media

comments

0%
20%
40%
60%

Remind
People

•  You’re
not
selling
anything

–  Par'cularly
on
purchase
intent
ques'ons

–  “We’re
not
asking
you
to
buy
something.
We’re
just
curious
if
you
have
any
plans.”

•  You
don’t
want
their
credit
card
info

•  You
don’t
need
them
to
download
anything

•  Your
registra9on,
cer9ﬁca9on
creden9als

Survey
Length

Survey
Length

•  this
a
long
survey
man
smh

•  That's
one
long
ass
survey

•  wow
that
was
a
long
survey

•  And
another
long
9me
bump!

•  This
survey
is
sooo
long
>.<

•  Taking
this
long
ass
survey.

•  This
Boring
Ass
Long
Survey..

•  Goba
do
this
long
ass
survey

•  This
survey
is
to
damn
long
..

•  Thisss
SURVEY
long
Ass
Hell
Man
!

•  Damnn
that
survey
was
long
as
shit

•  Omg
that
survey
thing
was
long
af'

•  Damnn
that
survey
was
long
as
shit

•  online
apps
with
long
ass
surveys<<

•  Taking
so
long
to
do
this
survey
ugh

•  I
got
to
do
these
2
long
ass
surveys

•  Filling
out
this
long
ass
survey.
<<

•  Taking
so
long
to
do
this
survey
ugh

•  Mothafn'
survey
was
long
as
shit.

•  This
survey
is
really
really
long
lol

•  Filled
out
that
long
ass
survey....
:/

•  This
survey
is
gonna
take
a
long
while

•  Dang
I
hate
doing
them
long
ahh
surveys

It’ll
Only
Take
Five
Minutes

•  SSI
survey
about
mobile
devices.
The
survey
says
1
mins,

but
it
will
take
much
longer
than
that.
Also,
survey
is

from
"qualtrics",
so
don't
speeding
through
the
survey
or

else
you
won't
get
credit.
btw,
another
purple
2
cents
ad

is
up

•  I
got
an
email
survey
that
said
it
would
take
1
min
so
I
did

it.
It
actually
took
about
20
-‐
they
made
me
watch
a

stupid
video
about
Audi
and
Le
Mans.

•  hate
surveys
that
say
"this
will
only
take
2
minutes"
and

end
up
taking
15
because
i
like
to
write
detailed

feedback

Lying

You
Should
Lie

•  Oh
my
god,
I
made
a
male
account
&
started
seeing
shitloads

of
surveys
compared
shoulda
done
that
sooner
:/

•  Just
lie
on
the
surveys.
Pretend
you
are
a
middle
aged

Hispanic
woman.
I
made
18,000
in
three
months.

•  Oh
hello
telephone
survey
people,
why
yes
my
name
is
monty

and
im
a
28yr
old
commercial
airline
pilot
with
a
phd.
Lol

•  I
love
to
answer
polls,
but
I
tend
to
lie
on
my
responses.

•  Do
I
lie
on
this
survey
or
do
I
be
truthful?
I
dont
wanna

incriminate
my
self...

27%
Confess
to
Lying*

*Not
confessing
does
not
equal
innocence

I
Lied
Because…

Answer

wasn’t

there

Trying
to

qualify

In
a
rush
Tired
Having

fun

Randoml
y
clicking

Upset

with

hos'ng

Don’t

know

I
Confess
Also…

• Par9als:
Do
not
read,
Speed,
Distracted,
Answer
on
behalf

• Poor
responders:
Don’t
admit
distrac9on,
Admit
to

random
clicking

Always
give
an
out

•  None
of
the
above

•  Never

•  Less
than
X

•  More
than
Y

•  Other

•  I
don’t
“do
that”

•  Please
specify:
_____________

Transparency

Why
demographics?

•  Every9me
a
survey
is
done
they
always
ask
if

ure
hispanic
like
why
does
it
fn
maber!

#whooocares

•  Why
do
surveys
ask
your
race
and
include
the

op9on
"other"?
What
the
f
could
other
be?

"Me?
I'm
actually
a
bellydancing
cactus."

Why
Screening?

•  WHY?

–  Survey
just
asked
me
where
am
I
living,
and
screened

me
out
immediately
when
I
said
East
Midlands

–  I
was
doing
a
survey
for
the
trailer
and
it
looks
boring

AF.
I
gave
my
honest
opinion
and
because
it
was
all

nega9ve
they
screened
me
out.

•  WHY
NOW?

–  just
got
10
mins
through
a
15
min
survey
only
to
be

disqualiﬁed....
Ggrrrrrrr

–  Got
a
survey
on
ﬁlms,
DON'T
COMPLETE
IT!
I
got
to
end
and

screened
me
out,
losing
20
mins
of
9me
Survey#
266616

–  Fed
up
of
doing
15
minutes
of
a
survey
then
gesng
screened

out,
why
can't
they
ask
the
relevant
ques9ons
at
the

beginning

I
don’t
understand
you

• Par9als
do
NOT
understand

• Poor
responders
do
NOT
understand

Demographics

•  Useless

– “To
classify
responses”

– “To
categorize
the
data”

•  Useful

– “For
example,
to
understand
how
men
and

women,
or
older
and
younger
people
diﬀer

– “To
ensure
we
listen
to
many
diﬀerent
types
of

people”

Screening

•  Screen
people
out
ASAP

•  Take
the
extra
minute
and
tell
people
why

– Those
are
all
the
ques9ons
we
have
for
you
today.

– We
are
sorry
to
screen
you
out.
However,
we

have
already
heard
from
a
number
of
people
with

characteris9cs
like
yours
and
need
to
make
sure

we
also
listen
to
people
with
other
characteris9cs

Screening

•  Let
people
know
they
may
be
screened

because
they
DO
use
or
product
or
because

they
do
NOT
use
a
product

•  Enforce
recruitment
and
invita9on
caps
so

that
certain
groups
don’t
get
far
more
surveys

than
other
groups

– You
know
that
women
get
fewer
invites
so
recruit

less
of
them

Incen'ves
Pay
for
Boredom

They
EARN
Incen'ves

•  ugh
@
this
boring
ass
survey.

•  i
hate
ﬁlling
these
surveys
in..boring

•  Focus
group
today
good
money
but
so
boringggg

•  swagbucks
is
mindnumbingly
boring
got
me
9
bucks
lol

•  Done
with
that
Youth
Risk
Behavior
Survey
-‐_-‐
shit
boring!!

•  online
surveys
are
intensely
boring
but
I
want
the
Paypal
points...

•  Just
par9cipated
in
a
focus
group.
Boring
but
LOVE
the
incen9ves!
$
$

•  Tonight
was
the
most
boring
'live'
market
research
I
have
ever

done!!!!

•  UGh
this
survey
is
so
boring
but
I'm
making
ten
dollars
an
hour
so

its
kind
of
nice

Money
Ranks
Highest

• Par9als
rank
every
incen9ve
higher

1
3
5
7
9
Top

Rank

Bocom

Rank

Don’t
Be
Boring

•  Loosen
up
your
grammar

– 
“What
is
the
best
idea
you
can
think
of?”

Don’t
Be
Boring

•  Loosen
up
your
grammar

– 
“What
is
the
best
idea
you
can
think
of?”

•  Use
more
casual
language

– “Tell
me
why
you
like
it.”

Don’t
Be
Boring

•  Loosen
up
your
grammar

– 
“What
is
the
best
idea
you
can
think
of?”

•  Use
more
casual
language

– “Tell
me
why
you
like
it.”

•  Try
a
light
joke
or
two

– “May
the
survey
force
be
with
you”

Responder
Thoughts

•  I
absolutely
loved
this
survey!
Thank
you
for
wan9ng
my

opinion.
:)

•  I
have
enjoyed
taking
this
survey,
because
it
tells
me
that

some
company
actually
cares
about
improving
its
surveys.

•  I
have
to
admit
I
was
wondering
if
any
company,
was
or

would
ever
pay
aben9on
to
panelists
concerns
usually
it

always
about
proﬁt
declared
or
not,
but
it's
true..
But
it's

comfor9ng
and
showing
some
good
faith
to
me,
being

considered
and
that
this
company
has
seemed
to
no9ce

these
concerns
need
to
be
addressed..
Thank
you.

•  I'm
glad
to
see
that
you
might
have
some
concerns
about

people
not
being
completely
truthful
with
their
answers.
I

always
answer
the
ques9ons
truthfully.
However,

some9mes
the
ques9ons
are
asked
in
such
a
manner
that

none
of
the
above
really
apply.

Conclusions

•  We
think
we
are
being
transparent
but
we
are

NOT

Conclusions

•  We
think
we
are
being
transparent
but
we
are

NOT

•  We
can
be
honest
without
betraying
the

method

Conclusions

•  We
think
we
are
being
transparent
but
we
are

NOT

•  We
can
be
honest
without
betraying
the

method

•  Must
write
beber

Conclusions

•  We
think
we
are
being
transparent
but
we
are

NOT

•  We
can
be
honest
without
betraying
the

method

•  Must
write
beber

•  Must
not
interpret
data
as
perfect

THANK
YOU

2014
Pla9num
Sponsor

Annie Pettit
Chief Research Officer, Peanut Labs
VP Research Standards, Research Now
Canada

THANK
YOU

DIY SURVEY PROGRAMMING DIY POLLINGDIY SAMPLE

Annie Pettit - Festival of New MR - 2014

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Annie Pettit - Festival of New MR - 2014

Similar to Annie Pettit - Festival of New MR - 2014 (20)

More from Ray Poynter

More from Ray Poynter (20)

Recently uploaded

Recently uploaded (20)

Annie Pettit - Festival of New MR - 2014