This document discusses how increasing survey coverage to include hard-to-reach populations may impact response rates. It presents simulation results exploring the relationship between coverage propensity, response propensity, and potential bias. The key findings are that increasing coverage can lower response rates, but total bias may still decrease if coverage and response propensities are strongly positively related. Absolute bias is more likely to decrease when variables influencing coverage and response are correlated.

1. Are the Hard-to-Cover Also Less Likely to Respond?
AAPOR 2015
Stephanie Eckman
Frauke Kreuter

2. Motivation
 Reducing undercoverage means:
‐ Phone: calling cell numbers
‐ Face-to-face: including homeless, institutionalized
‐ Web: providing tablet, internet access
 Costly
 Are the additional people included disproportionately
nonresponders?
2

3. Examples of Trade-Off
 RDD + mobile phone surveys (AAPOR Cell Phone Task Force Report, 2010)
‐ Lower RR among mobile only HHs
 LISS online panel (Leenheer & Scherpenzeel, 2013)
 Screener experiment (Tourangeau, Kreuter & Eckman, 2012)
3
Condition Coverage Rate Recruit. Rate
Internet not provided 90% 84%
Internet provided ~ 100% 79%
Condition Coverage Rate Response Rate
Direct: “Is anyone there 35-55?” 32% 86%
Roster: Age of all adults in HH 45% 72%

4. Simulation Setup
 Coverage propensity CPi
‐ Propensity for case to be included on frame
‐ Determined by X variables:
‐ comfort with technology, attachment to HH
 Response propensity RPi
‐ Propensity for case to respond
‐ Determined by Z variables
‐ at-home patterns, privacy concerns, topic interest
4

5. Path Diagram
5
X
Z
Y
CP
RP

6. Path Diagram
6
X
Z
Y
CP
RP
ρ
βX
βZ
γX
γZ

7. Simulation Setup
 𝑋, 𝑍 ~ 𝑀𝑁 0,
1 𝝆
𝝆 1
 Y related to X, Z (𝑌 = 10 + 𝜷 𝑿 ∗ 𝑋𝑖 + 𝜷 𝒁 ∗ 𝑍𝑖 + 𝜖)
 Survey A:
‐ 𝐶𝑃𝐴 = 𝑓(𝑋, 𝜸 𝑿)
‐ 𝑅𝑃𝐴 = 𝑓 𝑍, 𝜸 𝒁
 Survey B:
‐ CP increased by κ
‐ RP unchanged
7

8. Simulation Setup
 𝑋, 𝑍 ~ 𝑀𝑁 0,
1 𝝆
𝝆 1
 Y related to X, Z (𝑌 = 10 + 𝜷 𝑿 ∗ 𝑋𝑖 + 𝜷 𝒁 ∗ 𝑍𝑖 + 𝜖)
 Survey A:
‐ 𝐶𝑃𝐴 = 𝑓(𝑋, 𝜸 𝑿)
‐ 𝑅𝑃𝐴 = 𝑓 𝑍, 𝜸 𝒁
 Survey B:
‐ CP increased by κ
‐ RP unchanged
8
Parameter Values
𝝆 0.2, 0.6, 1
-0.2, -0.6, -1
𝜷 𝑿 0.2, 2
𝜷 𝒁 0.2, 2
𝜸 𝑿 0.2, 2, -0.2, -2
𝜸 𝒁 0.2, 2, -0.2, -2
κ 1, 4, 10

9. Outcome Measures
 When we increase coverage (A →B), what happens?
 Coverage rate, RR
‐ Under what conditions do we see trade-off?
 𝑏𝑖𝑎𝑠 𝑦 𝑅 = 𝑦 𝑅 − 𝑦𝑝𝑜𝑝
= 𝑦 𝑅 − 𝑦𝑐𝑜𝑣 + 𝑦𝑐𝑜𝑣 − 𝑦𝑝𝑜𝑝
Nonresponse Undercoverage
Bias Bias
‐ Under what conditions will bias decrease (in absolute value)?
9

10. Results: Rates
10cov(X,Z) > 0; strong relationship between X, CP & Z, RP

11. Results: When Does Absolute Bias Decrease?
X
Z
Y
CP
RP
ρ
βX
βZ
γX
γZ
11
+

12. Results: When Does Absolute Bias Decrease?
X
Z
Y
CP
RP
ρ
βX
βZ
γX
γZ
12
+

13. Results: Bias
X
Z
Y
CP
RP
ρ
βX
βZ
γX
γZ
13
+

15. www.iab.de
stephanie.eckman@iab.de
Website: stepheckman.com
Thanks – comments & ideas welcome

16. Results: Rates
16

17. Results: Rates
17

18. Coverage, Response Rates for 2 Surveys
18
Case
Landline Only Landline + Mobile
Covered
Response
Propensity
Covered
Response
Propensity
1 Yes 0.48 Yes 0.48
2 Yes 0.60 Yes 0.60
3 No 0.15 Yes 0.15
4 No 0.36 Yes 0.36
5 No 0.26 Yes 0.26
Coverage Rate 40% 100%
Response Rate 54% 37%