Seriously mixed methods  Do they risk non-response and attrition?  Ben Anderson  Chimera, University of Essex
The Menu•   Why bother?•   The background•   The panel and its methods•   Who dropped out (and why)•   What can we learn? ...
Multiple methods - why bother?• Data Triangulation:     • Different data on the same individuals     • Different instrumen...
Other reasons• Interaction of research modes (and  researchers!)     • Leads to insights & innovation• Multiple methods = ...
But• Such methods may      • Increase respondent burden      • Increase fears of privacy and surveillance• Or conversely  ...
An example: BT’s Digital Living project                                          Quantitative                             ...
GB Longitudinal PanelDec 1998                     Dec 1999            Dec 2000     • GB surveys (2500 individuals in 999 h...
Wave 1 process• Conduct face to face survey (HoL)                                                                     up t...
Added complexities...• Wave 1 bias     • 100% of households to have a telephone     • 50% to have a personal computer• Boo...
Wave 2 & 3 process• Attempt re-contact                                                                     up to 3 months•...
Response rates (individuals)• Cross- sectional      Undefined                        Survey plus diary                    ...
What do we want to know?• Did the three experimental ‘treatments’  cause non-response?• To keep it simple:     • Consider ...
Pathways W1                      W2                  W3                                    79%                 Interviewed...
Wave 1 to wave 2 effects• Comparison of response rates                              No call Yes, call          Difference ...
Wave 1 to wave 3 effects• Comparison of response rates                                                               Diffe...
Comparison                                        • Wave 1-2 • Wave 1-3              www.essex.ac.uk/chimera
Multivariate analysis• Logistic approach  – P(x) at t = control variables/known effects +    treatments  – Where X is     ...
Known effects• Based on Lynn et al (2005)                         Refuse               Non-contact       In HoL?     Age  ...
W2 resultsVariable         w2 refusal w2 non-contact    w2 non-contact                            (ind)             (hh)Ag...
W3 results                   w3        w3 non-contact   w3 non-contact                   refusal   (ind)            (hh)Ag...
Comparison•   Statistically significant results only•   logit, cluster (household identifier) [stata]                     ...
Attrition    Variable                           b    Age                             -0.052***    MRS Code                ...
Conclusions I• Multi-method projects give you ‘better’ data• And ‘better’ results (see elsewhere)• But     • They are reso...
Conclusions II• Disappointingly:     • Qualitative interviews did not help prevent non-response       or attrition• BUT en...
Get the data• All 3 waves of the survey     • UK Data Archive SN = 4607     • Free to UK Data Archive subscribers for non-...
Thank you• benander@essex.ac.uk            www.essex.ac.uk/chimera
Upcoming SlideShare
Loading in...5
×

Non-response and attrition in a multi-method longitudinal household panel survey

469

Published on

Paper presented at the Methodology of Longitudinal Surveys International Conference, 12-14 July 2006, University of Essex, Colchester, UK

Published in: Education, Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
469
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Non-response and attrition in a multi-method longitudinal household panel survey"

  1. 1. Seriously mixed methods Do they risk non-response and attrition? Ben Anderson Chimera, University of Essex
  2. 2. The Menu• Why bother?• The background• The panel and its methods• Who dropped out (and why)• What can we learn? www.essex.ac.uk/chimera
  3. 3. Multiple methods - why bother?• Data Triangulation: • Different data on the same individuals • Different instruments and methods (qual, quant, administrative) • Cross-confirmation and validation• Respondents lie, they forget and they don’t care • Multiple methods can unravel some of this • Different views - different insights• Patterns (what?) and explanations (why?) www.essex.ac.uk/chimera
  4. 4. Other reasons• Interaction of research modes (and researchers!) • Leads to insights & innovation• Multiple methods = real life methods• Increasingly valued in policy & evaluation research • ‘rounded view’ www.essex.ac.uk/chimera
  5. 5. But• Such methods may • Increase respondent burden • Increase fears of privacy and surveillance• Or conversely • Develop stronger relationships between researchers and respondents • Increase respondent ‘attachment’ www.essex.ac.uk/chimera
  6. 6. An example: BT’s Digital Living project Quantitative Phone call records PC/Internet usage logs Surveys Time-use diaries Interviews Shadowing & Observation Digital Ethnography Rich contextual Qualitative picture www.essex.ac.uk/chimera
  7. 7. GB Longitudinal PanelDec 1998 Dec 1999 Dec 2000 • GB surveys (2500 individuals in 999 hh) • Call record capture (635 of 999 hh) • Internet logs (16 of 999 hh) • ‘Long conversations’ (37 of 999 hh) Wave 1 Wave 2 Wave 3• Qualified random sample (clustered)• Wave 1 interviews = CAPI• Wave 2 & 3 = CATI www.essex.ac.uk/chimera
  8. 8. Wave 1 process• Conduct face to face survey (HoL) up to 6 months • Leave time-use diary • Obtain permission to collect call records • Obtain permission to re-contact for next survey and ethnography • Implement call record capture• Decide ethnographic sample frame (ICT rich/poor; income rich/poor) • Select households from eligible pool (requires survey data) • Approach households for interview (via survey agency) • Interview and arrange re-interviews/shadowing etc• Decide logging sample frame (anyone with Win95!) • Select households from eligible pool (requires survey data) • Approach households • Send disk (self-installer) www.essex.ac.uk/chimera
  9. 9. Added complexities...• Wave 1 bias • 100% of households to have a telephone • 50% to have a personal computer• Boost sample at wave 2 • Original address file, random selection • To maintain sample size • CATI• Overall a rare if not unique beast! www.essex.ac.uk/chimera
  10. 10. Wave 2 & 3 process• Attempt re-contact up to 3 months• Conduct CATI survey • Post out time-use diary • Check permission to collect call records • Obtain permission to re-contact for next survey and follow- up interviews• Boost sample (wave 2 only) • recruit & interview as Wave 1 www.essex.ac.uk/chimera
  11. 11. Response rates (individuals)• Cross- sectional Undefined Survey plus diary Wave 1 1093 42% Wave 2 6 649 25% Wave 3 10 723 30% • (unweighted) Survey only Non-response 668 26% 273 10% 918 36% 391 15% 840 35% 321 13% Childrens diary 163 82 73 No childrens diary 125 220 208 Child under 9 286 289 231 Total sample size 2608 2555 2406 Interviews Diaries• Longitudinal (Always a child) Never 697 462 13% 697 1415 39% Wave 1 only 511 14% 480 13% • (unweighted) Wave 2 only 136 4% 106 3% Wave 3 only 197 5% 214 6% Waves 1 and 2 224 6% 172 5% Waves 2 and 3 365 10% 68 2% Waves 1 and 3 159 4% 138 4% Waves 1, 2 and 3 842 23% 303 8% www.essex.ac.uk/chimera
  12. 12. What do we want to know?• Did the three experimental ‘treatments’ cause non-response?• To keep it simple: • Consider w1 to w2 and w1 to w3 effects only • Ignore boost sample • Focus on – refusal and non-contact in responding households (excludes movers) – Non-contact (non-responding households) – Attrition www.essex.ac.uk/chimera
  13. 13. Pathways W1 W2 W3 79% Interviewed in all waves 48% W1 interviewees 61% 12% 12% ‘Non-response’ w3 ‘Non-response’ w2 35% 25% 72% Attrition after w1 www.essex.ac.uk/chimera
  14. 14. Wave 1 to wave 2 effects• Comparison of response rates No call Yes, call Difference Difference records records (call Difference (instrumen Wave 2 outcome % % Total % records) (qual) tation) Interview 39.43 48.01 43.83 8.58 -0.44 10.43 Refusal 14.06 15.36 14.72 1.3 -9.10 -10.97 No contact in a responding hh 2.83 1.35 2.07 -1.48 1.65 -2.26 No hh response 23.96 15.36 19.56 -8.6 -11.47 -7.41 Other 19.72 19.93 19.82 0.21 19.36 10.21 N 1273 1335 52 51 Chi sq 43.26*** 16.96** 10.71*• % of w1 interviewees www.essex.ac.uk/chimera
  15. 15. Wave 1 to wave 3 effects• Comparison of response rates Difference Difference No call Yes, call (call Difference (instrumentWave 3 outcome records % records % Total % records) (qual) ation)Interview 77.89 78.59 78.28 0.70 -6.42 -4.83Refusal 5.58 6.09 5.87 0.51 -1.79 -2.53No contact in a respondinghh 2.99 1.88 2.36 -1.11 1.71 4.49No hh response 11.75 10.78 11.21 -0.97 8.84 1.70Other 1.79 2.66 2.28 0.87 -2.33 1.17Total 502 640 24 29Chi sq 2.778 2.61 2.94• % of w1 and w2 interviewees www.essex.ac.uk/chimera
  16. 16. Comparison • Wave 1-2 • Wave 1-3 www.essex.ac.uk/chimera
  17. 17. Multivariate analysis• Logistic approach – P(x) at t = control variables/known effects + treatments – Where X is • Refusal at t (responding hh) • Non-contact at t (responding hh) • Non-contact at t (non-responding hh) • Attrition www.essex.ac.uk/chimera
  18. 18. Known effects• Based on Lynn et al (2005) Refuse Non-contact In HoL? Age Elderly Elderly & Young Y Income Lower Higher and/or employed Y Gender Men Y Education Less Y Composition singles singles Y Culture Ethnic minorities Y Mobility High mobility High mobility N Location Urban Urban N• In addition: – Technophobia (‘resonance’) – MRS code (AB, C1, C2, D,E) as proxy for wealth www.essex.ac.uk/chimera
  19. 19. W2 resultsVariable w2 refusal w2 non-contact w2 non-contact (ind) (hh)Age -0.017* -0.062** -0.043***MRS Code 0.257** 0.158 0.225**Gender (female) -0.582** -1.428* -0.258*Qualification 0.095 -0.208 -0.048levelSingle person -0.821 0.399Ethnic minority -0.505 0.125Technophobia 0.028 0.164 0.049Call records 0.079 -0.735 -0.351Qualitative -0.875 1.831 -0.793Internet logging -0.678 -1.163Constant -1.961** -1.527 -0.565Pseudo r sq 0.044 0.128 0.08Chi sq 38.56854 20.49012 70.00324N 1172 881 1243 • logit, cluster (household identifier) [stata], values = b www.essex.ac.uk/chimera
  20. 20. W3 results w3 w3 non-contact w3 non-contact refusal (ind) (hh)Age -0.02 -0.034 -0.026**MRS Code -0.06 -0.187 -0.181Gender -0.431 0.215 -0.149(female)Qualification 0.052 0.14 -0.016levelSingle person -0.175 0.206Ethnic minority 0.429 1.810** 0.459Technophobia 0.032 -0.162 -0.039Call records 0.445 -0.39 -0.001Qualitative -0.125 0.782 0.936Internet logging -0.243 1.234 0.326Constant -1.737* -0.778 0.298Pseudo r sq 0.022 0.106 0.035Chi sq 12.18588 34.6003 19.14218N 880 747 932• logit, cluster (household identifier) [stata], values = b www.essex.ac.uk/chimera
  21. 21. Comparison• Statistically significant results only• logit, cluster (household identifier) [stata] www.essex.ac.uk/chimera
  22. 22. Attrition Variable b Age -0.052*** MRS Code 0.175* Gender (female) -0.446*** Qualification level 0.004 Single person 0.005 Ethnic minority -0.122 Call records -0.313 Qualitative 0.055 Internet logging -1.566 Technophobia 0.064* Region (North) yorkshire & humberside 1.310** east midlands 0.357 east anglia 0.866 south east (excl. london) 1.089* south west 0.893 • Added region west midlands 0.516 north west 0.713 wales 0.802 scotland 0.656 • Call records variable ‘nearly’ greater london 1.322** significant (p = 0.066) Constant -0.403 Pseudo r sq 0.12 Chi sq 110.2714 N 1190• logit, cluster (household identifier) [stata] www.essex.ac.uk/chimera
  23. 23. Conclusions I• Multi-method projects give you ‘better’ data• And ‘better’ results (see elsewhere)• But • They are resource hungry (researcher and respondent time/load) • They are complex to manage and analyse • You have to be multi-disciplinary/multi-skilled • All the usual qual/quant bickering takes place• All of which are good reasons to do them www.essex.ac.uk/chimera
  24. 24. Conclusions II• Disappointingly: • Qualitative interviews did not help prevent non-response or attrition• BUT encouragingly • None of the ‘treatments’ were associated with non- response or attrition• So overall we should do this more often! www.essex.ac.uk/chimera
  25. 25. Get the data• All 3 waves of the survey • UK Data Archive SN = 4607 • Free to UK Data Archive subscribers for non-commercial research• Held at Chimera (may be in UKDA eventually): • Qualitative transcripts • Call records (disclosure issues) • Internet usage logs www.essex.ac.uk/chimera
  26. 26. Thank you• benander@essex.ac.uk www.essex.ac.uk/chimera

×