The Future of Survey Research March 2017

1. CLICK TO EDIT MASTER TITLE STYLE
THE PAST, PRESENT & FUTURE
OF TRADITIONAL SURVEY RESEARCH
MARCH 2017
ddutwin@ssrs.com | 484-840-4406 | @ddutwin
David Dutwin, Ph.D.
SSRS EVP & Chief Methodologist

2. CLICK TO EDIT MASTER TITLE STYLETHINGS THAT MAKE YOU GO…
• Metrics in survey research are no different than other fields: cost and quality…cost we
know but what about quality?
• Response rates have dropped substantially: canary in a coal mine?
• Are non-probability panels the future?
• Can telephone last?
• What the hell just happened (re: election)?!?!?
• Where is traditional survey research in 5, 10 years?
© S S R S | A L L R I G H T S R E S E R V E D 2

3. CLICK TO EDIT MASTER TITLE STYLEA MATTER OF PAPERS
• RQ #1: What exactly has happened to telephonic survey response in the past decade?
• “Trends in Telephone Outcomes, 2008 - 2015.” Survey Practice, (2016) (D. Dutwin, P.
Lavrakas).
• RQ #2: Has the answer to RQ #1 done anything to data quality?
• “Telephone Sample Surveys: Dearly Beloved or Nearly Departed? Trends in Survey Errors
in the Age of Declining Response Rates.” (under peer review, 2017) (D. Dutwin, T.
Buskirk).
• RQ #3: Where do we stand with regard to low response rate probability versus
nonprobability?
• “Apples to Oranges or Gala versus Golden Delicious? Comparing Data Quality of Non-
Probability Internet Samples to Low Response Rate Probability Samples.” Public Opinion
Quarterly, (Special Issue on the Future of Survey Research, in press 2017) (D. Dutwin, T.
Buskirk).
© S S R S | A L L R I G H T S R E S E R V E D 3

4. CLICK TO EDIT MASTER TITLE STYLE
FIRST, WHAT IS THE STORY OF TELEPHONE
RESPONSE IN THE LAST DECADE?
(PLEASE DO NOT GET TOO DEPRESSED…)

5. CLICK TO EDIT MASTER TITLE STYLEDECLINING RESPONSE RATES
0%
5%
10%
15%
20%
25%
30%
35%
40%
1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Response Rates, 1997 - 2015
ABC Pew CBS
© S S R S | A L L R I G H T S R E S E R V E D 5

6. CLICK TO EDIT MASTER TITLE STYLETRENDS IN TELEPHONE DISPOSITIONS
I N T H E A G E O F C E L L P H O N E S
Data Study Scrub LL
Scrub
Cell
First
Year
Last
Year
LL
Sample
Cell
Sample
ABC ABC Polls Biz Purge 2010-2015 None 2008 2015 259,677 188,177
Gallup Gallup Daily Tracking Surveys None None 2009 2015 18,490,017 14,465,292
GfK AP Polls Aug 2012 to present None 2009 2014 434,405 100,586
NBC NBC Polls 2012 2015 125,382 140,384
PSRAI Pew Omnibus Biz Purge None 2010 2015 285,708 165,711
Pew
Pew Internet & American Life
Polls
Biz Purge None 2007 2015 369,301 185,385
RTI Survey of Consumer Attitudes None None 2010 2013 197,878 432,149
SRBI Confidential Biz Purge
Inactive
2014+
2007 2014 280,880 85,329
SSRS SSRS Omnibus MSG ID+ None 2009 2015 696,688 622,684
© S S R S | A L L R I G H T S R E S E R V E D 6
Initial ask was of ABC, CBS, Gallup, GfK, ICF, Ipsos, Nielsen, NORC, ORC, Pew, RAND, RTI, SRBI, TNS, Westat

7. CLICK TO EDIT MASTER TITLE STYLE
50% 51% 51% 51% 52% 52%
59%
54%
0%
20%
40%
60%
80%
2008 2009 2010 2011 2012 2013 2014 2015
Refusal Rate: Landlines
4% Increase
48%
52%
55% 53% 53%
48% 46% 46%
0%
20%
40%
60%
80%
2008 2009 2010 2011 2012 2013 2014 2015
Refusal Rate: Cellphones
2% Decrease
TRENDS IN DISPOSITIONS
R E F U S A L S & C A L L B A C K S
© S S R S | A L L R I G H T S R E S E R V E D 7

8. CLICK TO EDIT MASTER TITLE STYLETRENDS IN DISPOSITIONS
N O A N S WE R / A N S WE R I N G M A C H I N E S
© S S R S | A L L R I G H T S R E S E R V E D 8
26%
32% 32% 31% 33% 34% 35% 36%
0%
20%
40%
60%
80%
2008 2009 2010 2011 2012 2013 2014 2015
NA/AM Rate: Landlines
10% increase (4% since 2009)
21%
33% 31%
34%
37%
42%
45% 45%
0%
20%
40%
60%
80%
2008 2009 2010 2011 2012 2013 2014 2015
NA/AM Rate: Cellphones
24% increase (14% since 2010)

9. CLICK TO EDIT MASTER TITLE STYLETRENDS IN DISPOSITIONS
N O N - WO R K I N G
© S S R S | A L L R I G H T S R E S E R V E D 9
28% 28%
30%
33% 35%
37%
41% 40%
0%
20%
40%
60%
2008 2009 2010 2011 2012 2013 2014 2015
NW Rate: Landlines
12% increase
39%
34% 34%
30% 29%
25% 24% 24%
0%
20%
40%
60%
2008 2009 2010 2011 2012 2013 2014 2015
NW Rate: Cellphones
15% decrease (10% since 2009)

10. CLICK TO EDIT MASTER TITLE STYLETRENDS IN DISPOSITIONS
Y I E L D
© S S R S | A L L R I G H T S R E S E R V E D 10
.08
.07 .07 .07
.06
.05
.03 .03
.00
.02
.04
.06
.08
.10
.12
2008 2009 2010 2011 2012 2013 2014 2015
Yield: Landlines
.06
.05
.05 .05
.04 .04 .04
.04
.00
.02
.04
.06
.08
.10
.12
2008 2009 2010 2011 2012 2013 2014 2015
Yield: Cellphones
 Decreased yield by a factor of 2.4 (2.0 since 2009)
 Have gone from 14 records per complete to 46
▲ Decreased yield by a factor of 1.3 (1.15 since 2009)
▲ Have gone from 19 records per complete to 25

11. CLICK TO EDIT MASTER TITLE STYLELANDLINE OWNERSHIP
© S S R S | A L L R I G H T S R E S E R V E D 11
0%
20%
40%
60%
80%
100%
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025
Percent of HH Cell Phone Only

12. CLICK TO EDIT MASTER TITLE STYLETELEPHONIC RESPONSE SINCE 2008
• Landline has no more than 10 years left in its lifetime, barring a change in trend
• Refusal rates have potentially “hit the ceiling”…
• …Possibly because people, more and more, just don’t pick up
• Non-working rates increasing precipitously on the landline, declining on cells
• Overall yield on landlines has increased dramatically, but only modestly for cell phones
• The good news, however: costs and response rates have flattened.
© S S R S | A L L R I G H T S R E S E R V E D 12

13. CLICK TO EDIT MASTER TITLE STYLE
SO DOES THIS MEAN TELEPHONE SURVEY
QUALITY HAS SUBSTANTIALLY DECLINED?

14. CLICK TO EDIT MASTER TITLE STYLEDATA
Sample Name Mode Sample Size
Response Rate
Formula
The BRFSS Telephone 6,118,156 CASRO/RR4
CBS Polls Telephone 168,826 RR1
ABC Polls Telephone 179,939 RR3
Pew Polls Telephone 213,191 RR3
The GSS In-Person 27,219 RR3
The NHIS In-Person 1,232,179 RR3
© S S R S | A L L R I G H T S R E S E R V E D 14
• All Pew, CBS, and ABC Polls (455 total polls), all BRFSS
• Studies span 1996 - 2015

15. CLICK TO EDIT MASTER TITLE STYLECOMPUTING THE PRIMARY METRICS
Consider the demographic cross tabulation of Race and Region producing a 4-by-4 table.
Taking the absolute value of the difference between the row percentages and the
corresponding benchmarks from CPS produces a total of 16 absolute bias measures.
(Distribution of Region within Race)
© S S R S | A L L R I G H T S R E S E R V E D 15
Race Midwest South West Northeast
White
Black
Other
Hispanic
4 absolute bias measures
4 absolute bias measures
4 absolute bias measures
4 absolute bias measures
Row Percentages
The average of these 16 bias
measures represents the
Mean Absolute Bias (MAB) of
Region within Race.
Repeating the calculations for each of the column percentages (Distribution of
Race within Region) yields the MAB for Race within Region.

16. CLICK TO EDIT MASTER TITLE STYLETRENDS IN UNWEIGHTED MEAN ABSOLUTE BIASES
© S S R S | A L L R I G H T S R E S E R V E D 16
0%
1%
2%
3%
4%
5%
6%
7%
8%
Unweighted Overall Mean Absolute Biases
ABC CBS Pew Brfss GSS NHIS

17. CLICK TO EDIT MASTER TITLE STYLETRENDS IN UNWEIGHTED MEAN ABSOLUTE BIASES
© S S R S | A L L R I G H T S R E S E R V E D 17
3.9%
3.8%
3.9%
3.8%
4.0%
4.0%
4.3%
4.4%
4.4% 5.0%
5.3%
5.8%
5.7%
5.8%
5.6%
5.5%
5.6%
5.3%
5.2%
4.8%
0%
1%
2%
3%
4%
5%
6%
7%
Unweighted Overall Mean Absolute Biases

18. CLICK TO EDIT MASTER TITLE STYLEWEIGHTED TRENDS IN MEAN ABSOLUTE BIAS
© S S R S | A L L R I G H T S R E S E R V E D 18

19. CLICK TO EDIT MASTER TITLE STYLEPOTENTIAL IMPACT OF CELL PHONES
I N T H E R D D S A M P L E S
© S S R S | A L L R I G H T S R E S E R V E D 19
0%
10%
20%
30%
40%
50%
60%
70%
Percent of Interviews Attained by
Cell Phone
ABC CBS Pew Brfss
4.0%
4.5%
5.0%
5.5%
6.0%
6.5%
0%
10%
20%
30%
40%
50%
60%
70%
Overlay of Cell Phone Share and
MAB
Share of Cell Phones MAB

20. CLICK TO EDIT MASTER TITLE STYLE
SO CELL PHONES ARE THE FUTURE…BUT
KIDS DON’T ANSWER THEM RIGHT?
THEY CANNOT POSSIBLY BE REPRESENTATIVE….

21. CLICK TO EDIT MASTER TITLE STYLECHANGES IN TELEPHONE SURVEY DEMOGRAPHICS
© S S R S | A L L R I G H T S R E S E R V E D 21
34% 33%
65%
36% 32%
26%
31% 34%
7%
Benchmark Cell Sample Landline
Sample
Age Distribution By Sample
18-34
35-54
55+
Missing
8% 8% 6%
15% 12%
8%
12% 12%
8%
66% 66%
79%
Benchmark Cell
Sample
Landline
Sample
Race Distribution By Sample
White Non-Hisp
Black Non-Hisp
Hispanic
Other/Mixed
Non-Hisp
Missing
Kennedy (Pew Research Data), 2014

22. CLICK TO EDIT MASTER TITLE STYLE
© SSRS | ALL RIGHTS RESERVED 22
MEAN ABSOLUTE BIASES FOR CELL PHONE OWNERS COMPARED TO U.S.
ADULT POPULATION ON 12 DEMOGRAPHIC CROSS-TABULATIONS
MeanAbsoluteBias(fromtotalU.S.AdultPopulation)

23. CLICK TO EDIT MASTER TITLE STYLE
HOW DOES LOW RESPONSE RATE
TELEPHONE COMPARE TO
NONPROBABILITY RESEARCH IN TERMS
OF DATA QUALITY?

24. CLICK TO EDIT MASTER TITLE STYLEPROBABILITY VS. NONPROBABILITY
0% 2% 4% 6% 8% 10% 12% 14% 16% 18%
Web 1 Unweighted
Web 2 Unweighted
SSRS Omnibus Cell Phones Unweighted
SSRS Telephone Omnibus Unweighted
SSRS Tracker Telephone Unweighted
NHIS Unweighted
Web 1 Raked
Web 2 Raked
Web 2 Propensity Weighted
Web 2 Propensity Weighted and Raked
Web 2 Matched
Web 2 Matched and Raked
Web 1 Matched
Web 1 Matched and Raked
Cell Phones Raked
SSRS Omnibus Telephone Raked
SSRS Tracker Raked
NHIS Weighted
Mean Bias of Interactive Marginals by Sample/Weighting Type
Race within Education
Age within Education
Region within Education
Race within Age
Education within Age
Region within Age
Age within Race
Region within Race
Education within Race
Race within Region
Age within Region
Education within Region
© S S R S | A L L R I G H T S R E S E R V E D 24

25. CLICK TO EDIT MASTER TITLE STYLEOVERALL AVERAGE BIASES COMPARISON
© S S R S | A L L R I G H T S R E S E R V E D 25

26. CLICK TO EDIT MASTER TITLE STYLEOVERALL VARIANCE OF THE BIASES COMPARISON
© S S R S | A L L R I G H T S R E S E R V E D 26

27. CLICK TO EDIT MASTER TITLE STYLEPROBABILITY VS. NONPROBABILITY
© S S R S | A L L R I G H T S R E S E R V E D 27
 Malhotra & Krosnick (2007), Chang & Krosnick (2009), Dutwin and Buskirk (2015), Tourangeau,
Conrad, and Couper (2013), Walker et al. (2009), Yeager et al. (2011)…

28. CLICK TO EDIT MASTER TITLE STYLE
 The Advertising Research Foundation
(ARF) set up the Online Research
Quality Council (ORQC) in August 2007.
 17 US online panel providers (all using
nonprobability samples) a telephone
sample panel, and a mail sample panel.
 Factual and behavioral questions were
asked with the same question wording
as the benchmarks they would be
compared against; data weighted.
 Wide variation across panels in the
survey estimates.
 Sample tenure effected estimates
derived from panels.
ARF STUDY
© S S R S | A L L R I G H T S R E S E R V E D 28
18
19
20
20
21
23
24
26
26
27
27
28
28
30
30
31
32
33
Benchmark
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
Comparison of Smoking Prevalence; Benchmark
vs. 17 Opt-In Panels
Walker, R., Pettit, R., & Rubinson, J. (2009). A special report from the
Advertising Research Foundation: The foundations of quality
initiative: A five-part immersion into the quality of online research.
Journal of Advertising Research, 49, 464–485.

29. CLICK TO EDIT MASTER TITLE STYLE
SEGUE…SO WHAT ABOUT THE
ELECTION ANYWAY?

30. CLICK TO EDIT MASTER TITLE STYLETHE STORY OF THE 2016 ELECTION AS WE KNOW IT
0%
1%
2%
3%
4%
5%
6%
7%
8%
2000 2002 2004 2006 2008 2010 2012
Accuracy in U.S. Polls, 2000 - 2012
Gubernatorial
Senatorial
House
Presidential
© S S R S | A L L R I G H T S R E S E R V E D 30

31. CLICK TO EDIT MASTER TITLE STYLETHE STORY OF THE 2016 ELECTION AS WE KNOW IT
The Macro Problem:
• Polls always challenging: a survey of a population that does not yet exist.
• Polls ask if election were held today.
• Polls try to predict the popular vote…so what is the problem here?
Still There Were Problems: State polls in aggregate, and “the aggregators,” can predict the
Electoral College and did not:
• State polls typically of lower cost: RBS sample; fewer call attempts; smaller sample size; less
sophisticated likely voter models employed
• State polls done less frequently
Possible Causes of Poor Election Results:
• Momentum effect
• People lying about intention
• Undecideds/3rd party intenders going Trump/Last minute decision-making
• Nonresponse error
• Likely voter model wrong
• Challenges to state polling
© S S R S | A L L R I G H T S R E S E R V E D 31

32. CLICK TO EDIT MASTER TITLE STYLETHE STORY SO FAR
The Past
• Survey response down…telephonic survey costs up…but
 Mostly in landlines
 Cellphone response and costs flat since 2014
 Data bias up starting in ‘05 but on a downward since
‘09, little net effect
The Present
• The age of fragmentation and fit for purpose…telephone,
abs, nonprobability, probability panel, omnibus survey…is
already upon us.
• There remains a strong correlation between cost and
data quality…period.
What will the future hold?
© S S R S | A L L R I G H T S R E S E R V E D 32

33. CLICK TO EDIT MASTER TITLE STYLEFUTURE 1
This presentation has established:
• Landline future is limited
• Cellphone data has very low bias, only marginally worse than in-person
• Cellphone costs per interview have flattened, as have response rates
The next few years are critical, but the data suggest we all may be surprised at the future of
telephone research
© S S R S | A L L R I G H T S R E S E R V E D 33

34. CLICK TO EDIT MASTER TITLE STYLEFUTURE 2: BETTER OPT-IN DATA
Many people have been working hard to find a way to make opt-in data quality on its
own better. How:
• Webographics and other propensity adjustments.
• Sample matching: a potential way to stack the deck before data collection.
• Advanced calibration: in short, rake to everything.
• Balanced samples: pay the extra money to get samples pre-balanced on key
demographics.
…In three years of many people trying these out, so far the results have been inconsistent.
© S S R S | A L L R I G H T S R E S E R V E D 34

35. CLICK TO EDIT MASTER TITLE STYLEFUTURE 3: HYBRIDS
Fielding studies that are a blend of probability and non-probability samples.
• The goal: cost savings without significant risk of bias.
• Can use propensity or calibration techniques to combine.
• Two approaches: 1) true hybrid 2) limited questions in probability for calibration purposes
only.
• Must be mindful of mode effects.
…In three years of many people trying these out, so far the results have been somewhat
promising, but inconsistent.
© S S R S | A L L R I G H T S R E S E R V E D 35

36. CLICK TO EDIT MASTER TITLE STYLECASE STUDY:
C A L I B R A T I O N T O P R O B A B I L I T Y E S T I M A T E S
© S S R S | A L L R I G H T S R E S E R V E D 36
Benchmark Unweighted Raked Propensity Calibration1 Calibration 2
Moved last 6 months 7% 14% 10% 9% 9% 9%
One person household 23% 18% 18% 18% 17% 19%
Do not own a computer 16% 6% 7% 7% 10% 16%
Do not own a smartphone 32% 28% 28% 28% 29% 31%
Do not own a tablet 62% 53% 42% 59% 58% 60%
Own home 67% 63% 57% 58% 67% 67%
Single 28% 30% 33% 32% 28% 28%
Not employed 42% 42% 46% 45% 45% 45%
Have children 37% 39% 35% 35% 36% 36%
38%
47%
27%
36%
42%
27%27%
31%
15%17%
22%
15%
7% 9% 11%
0%
10%
20%
30%
40%
50%
Overall Bias w/o OH & S w/o OH, S & C
Bias in 8 Estimates Unweighted Raked Propensity Calibration1 Calibration 2

37. CLICK TO EDIT MASTER TITLE STYLESOMETIMES HYBRIDS WORK…
© S S R S | A L L R I G H T S R E S E R V E D 37
Hybrid: anything derived from heterogeneous sources, or composed of elements of
different or incongruous kinds.

38. CLICK TO EDIT MASTER TITLE STYLE…AND SOMETIMES THEY DON’T
© S S R S | A L L R I G H T S R E S E R V E D 38

39. CLICK TO EDIT MASTER TITLE STYLEFUTURE 3: PROBABILITY PANELS?
Mimicking Opt-In Panels, but with a Probability Recruitment Method
• The goal: cost savings without significant risk of bias.
• About half the cost of custom telephone.
• However: typically small panel sizes (10k – 50k) limit consistent tracking or low incidence
studies.
• Suffer from the same non-response patterns of opt-in samples.
© S S R S | A L L R I G H T S R E S E R V E D 39

40. CLICK TO EDIT MASTER TITLE STYLEDATA QUALITY OF PROBABILITY PANELS
© S S R S | A L L R I G H T S R E S E R V E D 40

41. CLICK TO EDIT MASTER TITLE STYLEFUTURE 4: ADDRESS BASED DESIGNS?
High quality policy researchers do not have an “in-person” budget, but are searching
for telephone alternatives
• Tendency to systematic bias typical of telephones, if not worse.
• High response rates often require multiple modes
• Multiple languages more challenging
• Need field time to do well
© S S R S | A L L R I G H T S R E S E R V E D 41

42. CLICK TO EDIT MASTER TITLE STYLETHE FUTURE? ITS COMPLICATED
• One size does not fit all; we are truly in the “fit for purpose” age.
• The “space race” to “fix” nonprobability will continue. Every study will need its own model
• Telephone will continue to serve about as well as they have for the past 10 years, if, with a
higher level of distrust and elevated costs
• ABS will serve where applicable; in-person will continue to be the method of choice for
those with deep pockets
• Hybrid surveys will find a greater role in industries where future telephone costs are
unpalatable
© S S R S | A L L R I G H T S R E S E R V E D 42

43. CLICK TO EDIT MASTER TITLE STYLE
DAVI D DUT WI N
4 8 4 . 8 4 0 . 4 4 0 6
ddutwin@ssrs.com
@ddutwin
@ssrs_solutions | 484.840.4300 | info@ssrs.com
CONTACT ME
WI T H A N Y Q U E S T I O N S