Undercoverage plaques many frames - housing units are missed by listers or do not appear on the postal service list; persons with tenuous connections to households are not captured in rosters; persons hide their eligibility during screener interviews. The literature on undercoverage suggests several methods for improving the coverage of such frames, via a missed housing unit procedure, or detailed probes about household members, or disguising the target population in survey questions. However, each of these solutions introduces additional costs into the survey process. In this way, survey designers face a coverage-cost trade-off. In addition, there is increasing evidence that the cases found via these coverage-improvement measures are disproportionately nonresponders to the survey request. Thus there appears to be a coverage-nonresponse trade-off as well. Together these points raise the question of how much effort we should put into increasing coverage, when such efforts increase costs and nonresponse? This presentation will review empirical evidence for these trade-offs and search for clues to the mechanisms underlying the connection between nonresponse and undercoverage.

1. The Coverage-Nonresponse Trade-Off
JSM 2014
Stephanie Eckman
Frauke Kreuter

2. Motivation
 Reducing undercoverage means:
‐ Adding cell numbers to RDD frame
‐ Including homeless, institutionalized in general population
surveys
‐ Providing tablet & internet access
 Efforts are costly
 Are the people included with these efforts
disproportionate nonresponders?
2

3. Examples
 RDD + mobile phone surveys
‐ Lower response rates among mobile only HHs
(AAPOR Cell Phone Task Force Report)
 LISS online panel
‐ Lower recruitment rates among cases without internet
(Leenheer & Scherpenzeel 2013)
 Random walk
‐ Interviewers may skip HHs that look like NRs (Alt et al 1991,
Manheimer & Hyman 1949 )
 Half open interval procedure
‐ Interviewers may fail to cover units that look like NRs (Eckman
& O’Muircheartaigh 2011)
3

4. Screening Study
 2 versions of screener questions
‐ Direct “Is anyone in this household 35-55?”
‐ Full HH roster Age of all adults in HH
Condition
Screener
Completion
Rate
Eligibility
Rate
Interview
Completion
Rate
Response
Rate Yield
Direct 59.3 31.8 86.3 32.3 285
Roster 53.5 45.1 71.5 23.9 361
 Trade-off: High response rate or high coverage?
Tourangeau, Kreuter & Eckman 2013 4

5. Mechanisms Behind Trade-Off
 Respondent side
‐ Burden
‐ Learning to use internet/computer difficult
‐ Survey on cell phone annoying
‐ Hidden refusals: Respondents screen out rather than refuse
 Interviewer side
‐ Judged by response rate, not coverage
‐ Coverage & response are different skill sets
5

6. Choice Faced by Survey Designers
6
 Choose A or B?
 High RR in A hides low
coverage rate
 Cost considerations
Coverage
Rate
High Low
Response Rate
High A
*
Low B

7. Screening Study
 Coming back to this example:
‐ Direct: high RR, low coverage
‐ Roster: low RR, high coverage
Condition
Screener
Completion
Rate
Eligibility
Rate
Interview
Completion
Rate
Response
Rate Yield
Direct 59.3 31.8 86.3 32.3 285
Roster 53.5 45.1 71.5 23.9 361
Tourangeau, Kreuter & Eckman (2013) 7

8. Propensity Models
 Logit models, run separately for 2 screener conditions
 IVs: female, age (squared), # calls (squared), party identification,
postal code, interviewer payment, refusal flag, mobile indicator
 Coverage Model
‐ Case base: expected to be eligible (n=2,904)
‐ DV: screened & eligible
‐ Pseudo-R2: 15% direct; 10% roster
 Response Model
‐ Case base: found to be eligible (n=735)
‐ DV: complete main interview
‐ Pseudo-R2: 20% direct; 33% roster
8

9. Propensity Models
 Logit models, run separately for 2 screener conditions
 IVs: female, age (squared), # calls (squared), party identification,
postal code, interviewer payment, refusal flag, mobile indicator
 Coverage Model
‐ Case base: expected to be eligible (n=2,904)
‐ DV: screened & eligible
‐ Pseudo-R2: 15% direct; 10% roster
 Response Model
‐ Case base: found to be eligible (n=735)
‐ DV: complete main interview
‐ Pseudo-R2: 20% direct; 33% roster
9
}}
CPdirect
CProster
RPdirect
RProster

10. Coverage Propensities
Correlation 0.79 10

11. Response Propensities
Correlation 0.88 11

12. Overall Inclusion Propensity
Correlation 0.86 12

13. Overall Inclusion Propensity
Correlation 0.86 13

14. First Look at Bias
14

15. Research Agenda
 Mechanisms of connection between them
‐ Interviewers? Respondents?
 Bias due to undercoverage & NR
 How much should we spend to increase coverage,
if it only increases nonresponse?
15

16. www.iab.de
Thanks – comments & ideas welcome
stephanie.eckman@iab.de
Website: stepheckman.com

17. References
17
Alt, C., Bien, W. & Krebs, D. (1991). “Wie zuverlässig
ist die Verwirklichung von Stichprobenverfahren?
Random route versus
Einwohnermeldeamtsstichprobe”. ZUMA
Nachrichten, 28, 65-72.
American Association for Public Opinion Research
(2010). “Cell Phone Task Force Report: New
Considerations for Survey Researchers When
Planning and Conducting RDD Telephone Surveys in
the U.S. With Respondents Reached via Cell Phone
Numbers”.
Eckman, S. & O’Muircheartaigh, C. (2011).
“Performance of the Half-Open Interval Missed
Housing Unit Procedure”. Survey Research Methods,
5(3), 125-131.
Hainer, P. (1987). “A Brief and Qualitative
Anthropological Study Exploring the Reasons for
Census Coverage Error Among Low Income Black
Households”. In Report prepared under contract
with the Census Bureau.
Leenheer, J. & Scherpenzeel, A. (2013). “Does it
Pay Off to Include Non-Internet Households in an
Internet Panel?” International Journal of Internet
Science 8 (1), 17–29.
Manheimer, D. & Hyman, H. (1949). “Interviewer
Performance in Area Sampling”. The Public
Opinion Quarterly, 13(1), 83-92.
Tourangeau, R., Kreuter, F. & Eckman, S. (2012).
"Motivated Underreporting in Screening
Interviews“. Public Opinion Quarterly 76(3), 453-
469.

18. Coverage & Nonresponse Propensities
Correlation 0.43 18

19. Coverage & Nonresponse Propensities
Correlation 0.67 19

20. www.iab.de
Thanks – comments & ideas welcome
stephanie.eckman@iab.de
Website: stepheckman.com