The Future of Traditional Survey Research

the future of
traditional survey research
research. refined.
ssrs
David Dutwin, PhD
EVP & Chief Methodologist
@ddutwin
484-840-4406
ddutwin@ssrs.com
February 1-3 2016

2
ssrsresearch. refined.
what is “traditional” survey research?
© ssrs | all rights reserved

3
why now? why this presentation?
2015 was a good year for research on research
• pace of research on non-probability samples exploding
• finally, some recent data on telephone surveys
• questions on trusting polling
• people need answers!
– AAPOR future of telephone surveys
– collaborative data science R&D
– Dutwin/Buskirk Bias Project

4
92%
25%
53% 52%
26%
22%
78%
24%
44%
38%
25%
16%
82%
27%
41%
30%
22%
11%
88%
44%
39%
32%
24%
11%
Online Mobile CATI Face-to-Face CAPI Mail
trends in survey research methods
source: greenbook research industry trends reports 2012 - 2015
2012 2013 2014 2015
the trends

research. refined.
ssrs
telephone
(yes, telephone)

6
0%
20%
40%
60%
80%
100%
percent of household cell phone only
source: CDC NHIS wireless substitution reports
landline ownership on borrowed time

7
0%
5%
10%
15%
20%
25%
30%
35%
40%
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
response rates 1995-2013
source: reported by multiple vendors
ABC Pew CBS
telephone response rates

9
trends in dispositions: refusals & callbacks
62%
66% 79%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
2007 2008 2009 2010 2011 2012 2013 2014 2015
refusals and callbacks
Cell Landline

10
trends in dispositions: AM/NA
29%
58%
31%
52%
0%
10%
20%
30%
40%
50%
60%
70%
Date 2007 2008 2009 2010 2011 2012 2013 2014
answering machines/no answers
Cell Landline

11
trends in dispositions: yield
19
49
15
45
0
10
20
30
40
50
60
2007 2008 2009 2010 2011 2012 2013 2014 2015
yields 2007-2015
Cell Landline

12
“there is no such thing as a probability sample anymore”
- Andrew Gelman and all who profit from making such a statement
the response
• the sampling is probabilistic
• non-response is very high: but to what degree is it systematic?
• research (Keeter, Groves, Dutwin, others), finds relatively very little systemization in
outcome measures; more in demographics, but that is correctable.

13
age distribution by sample
34% 33%
65%
36% 32%
26%
31% 34%
7%
Benchmark Cell Sample Landline
Sample
18-34
35-54
55+
Missing
race distribution by sample
8% 8%
6%
15% 12%
8%
12% 12%
8%
66% 66%
79%
Benchmark Cell Sample Landline
Sample
White Non-Hisp
Black Non-Hisp
Hispanic
Other/Mixed
Non-Hisp
Missing
changes in telephone survey demographics
Source: Pew Research Center 2014

14
cell phone interviewers
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
2006 2007 2008 2009 2010 2011 2012 2013 2014
percent cell phone interviews 2005 - 2013
SCA BRFSS CBS Pew ABC

15
trends in dispositions: yield
0
1
2
3
4
5
6
7
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
mean absolute error by sample type 1995–2013
Telephone Unweighted Telephone Weighted

research. refined.
ssrs
online
probability panels

17
why online?
• inexpensivecost
• not for message testing or other embedded
experiments
• not for trending
• yes, for point estimates
nonprobabilistic
…does it matter?
• 12-15% not online
some people aren’t
online
…does it matter?

18
data: Dutwin & Buskirk 2015
0 5 10 15 20
Luth Web Unweighted
Research Now Web Unweighted
SSRS Omnibus Cell Phones Unweighted
SSRS Telephone Omnibus Unweighted
SportsPoll Telephone Unweighted
NHIS Unweighted
Luth Web Raked
Research Now Web Raked
Research Now Propensity Weighted
Research Now Propensity Weighted and Raked
Research Now Matched
Research Now Matched and Raked
Luth Web Matched
Luth Web Matched and Raked
Cell Phones Raked
SSRS Omnibus Telephone Raked
SportsPoll Raked
NHIS Weighted
Mean Absolute Biases for each of the 12 Sets of Row/Column Percentages by
Sample/Weighting Type
Race within Education
Age within Education
Region within Education
Race within Age
Education within Age
Region within Age
Age within Race
Region within Race
Education within Race
Race within Region
Age within Region
Education within Region

19
2.2%
2.8%
3.9%
2.7%
3.9%
2.2%
1.3%
1.6%
1.5%
1.8%
0.8%
Luth Web Raked
Luth Web Matched
Centris ABS Raked
Cell Phones Raked
SportsPoll Raked
NHIS Weighted
mean bias of nested demographics by weighted/matched samples
5.9%
6.9%
6.7%
3.4%
3.0%
2.7%
2.2%
Luth Web Unweighted
Centris ABS Unweighted
NHIS Unweighted
mean bias of interactive marginal by unweighted samples
Unequal Weighting Effects: Phone: 1.21, Web Rake: 2.83. Web Propensity: 6.35, Web Propensity and Rake: 5.43

20
1.3%
2.4%
2.5%
2.5%
1.6%
1.0%
0.6%
0.7%
0.8%
0.7%
0.2%
Luth Web Raked
Luth Web Matched
Centris ABS Raked
Cell Phones Raked
SportsPoll Raked
NHIS Weighted
standard deviation of the biases in weighted samples
3.1%
5.7%
2.4%
1.4%
1.3%
0.8%
0.9%
Luth Web Unweighted
Centris ABS Unweighted
NHIS Unweighted
standard deviation of the biases in unweighted samples
Unequal Weighting Effects: Phone: 1.21, Web Rake: 2.83, Web Propensity: 6.35, Web Propensity and Rake: 5.43, Web Matched: 1.18

21
the future of web panels: modelling
currently researchers try to make nonprobability panels
more representative via sample matching, propensity
modelling, and calibration
to date, each has had very mixed success (as evidenced
by the immediately previous slides)
that said, researchers on racing to find ways to make
such samples pseudo-probabilistic

research. refined.
ssrs the cell phone
as an interviewing device

23
web surveys on cell phones
Currently it is illegal to text phones without prior consent of the user
• the cell phone is close to becoming a near universal-coverage sampling frame
• probability panels are becoming more common
• recent research finds that 25-45% of persons conducting web surveys are
completing them on a smart phone
– thus in the future one can conceive of using cell phone numbers as a sampling
frame, but surveys being conducted via smartphone web browsers or apps
– thus in the present, interviewing software providers must have strong solutions for
adaptive design

24
web surveys on cell phones

25
surveying cell phone data
challenges
• many challenges
with regard to the
ability of various
OS having
differential
reporting
capabilities
data gathering
• GPS and
cameras serve as
data gathering
devices and
prompts to initiate
surveys
solutions
• solutions include
app-like survey
platforms and
actual app
platform surveys
limitations
• limitations on
apps reporting
use

research. refined.
ssrs
the future

27
the age of fragmentation
telephones/ABS
• still the leader in data quality
• costs continue to increase, will it end?
• pushing the market to omnibus and probability panel solutions
online panels
• inexpensive and as accurate as not
• every panel is different
• the race is on to make them pseudo-probabilistic
smartphones
• penetration still increasing
• some in the industry fully up to speed on survey design, others not so much
big data
• limited utility to replace traditional survey research, despite claims to the contrary
• still, highly useful when the research question matches the available data
other approaches
• many other approaches available, that again are highly useful given a particular research question
• eye-tracking; gamification; biometric response; wearables; internet of things

David Dutwin, PhD
EVP & Chief Methodologist
@ddutwin
484-840-4406
ddutwin@ssrs.com
research. refined.
ssrs
thank you
for your time

The Future of Traditional Survey Research

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to The Future of Traditional Survey Research

Similar to The Future of Traditional Survey Research (20)

More from SSRS Market Research

More from SSRS Market Research (20)

Recently uploaded

Recently uploaded (20)

The Future of Traditional Survey Research