Your SlideShare is downloading. ×
0
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Non-response and attrition in a multi-method longitudinal household panel survey
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Non-response and attrition in a multi-method longitudinal household panel survey

450

Published on

Paper presented at the Methodology of Longitudinal Surveys International Conference, 12-14 July 2006, University of Essex, Colchester, UK

Paper presented at the Methodology of Longitudinal Surveys International Conference, 12-14 July 2006, University of Essex, Colchester, UK

Published in: Education, Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
450
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Seriously mixed methods Do they risk non-response and attrition? Ben Anderson Chimera, University of Essex
  • 2. The Menu• Why bother?• The background• The panel and its methods• Who dropped out (and why)• What can we learn? www.essex.ac.uk/chimera
  • 3. Multiple methods - why bother?• Data Triangulation: • Different data on the same individuals • Different instruments and methods (qual, quant, administrative) • Cross-confirmation and validation• Respondents lie, they forget and they don’t care • Multiple methods can unravel some of this • Different views - different insights• Patterns (what?) and explanations (why?) www.essex.ac.uk/chimera
  • 4. Other reasons• Interaction of research modes (and researchers!) • Leads to insights & innovation• Multiple methods = real life methods• Increasingly valued in policy & evaluation research • ‘rounded view’ www.essex.ac.uk/chimera
  • 5. But• Such methods may • Increase respondent burden • Increase fears of privacy and surveillance• Or conversely • Develop stronger relationships between researchers and respondents • Increase respondent ‘attachment’ www.essex.ac.uk/chimera
  • 6. An example: BT’s Digital Living project Quantitative Phone call records PC/Internet usage logs Surveys Time-use diaries Interviews Shadowing & Observation Digital Ethnography Rich contextual Qualitative picture www.essex.ac.uk/chimera
  • 7. GB Longitudinal PanelDec 1998 Dec 1999 Dec 2000 • GB surveys (2500 individuals in 999 hh) • Call record capture (635 of 999 hh) • Internet logs (16 of 999 hh) • ‘Long conversations’ (37 of 999 hh) Wave 1 Wave 2 Wave 3• Qualified random sample (clustered)• Wave 1 interviews = CAPI• Wave 2 & 3 = CATI www.essex.ac.uk/chimera
  • 8. Wave 1 process• Conduct face to face survey (HoL) up to 6 months • Leave time-use diary • Obtain permission to collect call records • Obtain permission to re-contact for next survey and ethnography • Implement call record capture• Decide ethnographic sample frame (ICT rich/poor; income rich/poor) • Select households from eligible pool (requires survey data) • Approach households for interview (via survey agency) • Interview and arrange re-interviews/shadowing etc• Decide logging sample frame (anyone with Win95!) • Select households from eligible pool (requires survey data) • Approach households • Send disk (self-installer) www.essex.ac.uk/chimera
  • 9. Added complexities...• Wave 1 bias • 100% of households to have a telephone • 50% to have a personal computer• Boost sample at wave 2 • Original address file, random selection • To maintain sample size • CATI• Overall a rare if not unique beast! www.essex.ac.uk/chimera
  • 10. Wave 2 & 3 process• Attempt re-contact up to 3 months• Conduct CATI survey • Post out time-use diary • Check permission to collect call records • Obtain permission to re-contact for next survey and follow- up interviews• Boost sample (wave 2 only) • recruit & interview as Wave 1 www.essex.ac.uk/chimera
  • 11. Response rates (individuals)• Cross- sectional Undefined Survey plus diary Wave 1 1093 42% Wave 2 6 649 25% Wave 3 10 723 30% • (unweighted) Survey only Non-response 668 26% 273 10% 918 36% 391 15% 840 35% 321 13% Childrens diary 163 82 73 No childrens diary 125 220 208 Child under 9 286 289 231 Total sample size 2608 2555 2406 Interviews Diaries• Longitudinal (Always a child) Never 697 462 13% 697 1415 39% Wave 1 only 511 14% 480 13% • (unweighted) Wave 2 only 136 4% 106 3% Wave 3 only 197 5% 214 6% Waves 1 and 2 224 6% 172 5% Waves 2 and 3 365 10% 68 2% Waves 1 and 3 159 4% 138 4% Waves 1, 2 and 3 842 23% 303 8% www.essex.ac.uk/chimera
  • 12. What do we want to know?• Did the three experimental ‘treatments’ cause non-response?• To keep it simple: • Consider w1 to w2 and w1 to w3 effects only • Ignore boost sample • Focus on – refusal and non-contact in responding households (excludes movers) – Non-contact (non-responding households) – Attrition www.essex.ac.uk/chimera
  • 13. Pathways W1 W2 W3 79% Interviewed in all waves 48% W1 interviewees 61% 12% 12% ‘Non-response’ w3 ‘Non-response’ w2 35% 25% 72% Attrition after w1 www.essex.ac.uk/chimera
  • 14. Wave 1 to wave 2 effects• Comparison of response rates No call Yes, call Difference Difference records records (call Difference (instrumen Wave 2 outcome % % Total % records) (qual) tation) Interview 39.43 48.01 43.83 8.58 -0.44 10.43 Refusal 14.06 15.36 14.72 1.3 -9.10 -10.97 No contact in a responding hh 2.83 1.35 2.07 -1.48 1.65 -2.26 No hh response 23.96 15.36 19.56 -8.6 -11.47 -7.41 Other 19.72 19.93 19.82 0.21 19.36 10.21 N 1273 1335 52 51 Chi sq 43.26*** 16.96** 10.71*• % of w1 interviewees www.essex.ac.uk/chimera
  • 15. Wave 1 to wave 3 effects• Comparison of response rates Difference Difference No call Yes, call (call Difference (instrumentWave 3 outcome records % records % Total % records) (qual) ation)Interview 77.89 78.59 78.28 0.70 -6.42 -4.83Refusal 5.58 6.09 5.87 0.51 -1.79 -2.53No contact in a respondinghh 2.99 1.88 2.36 -1.11 1.71 4.49No hh response 11.75 10.78 11.21 -0.97 8.84 1.70Other 1.79 2.66 2.28 0.87 -2.33 1.17Total 502 640 24 29Chi sq 2.778 2.61 2.94• % of w1 and w2 interviewees www.essex.ac.uk/chimera
  • 16. Comparison • Wave 1-2 • Wave 1-3 www.essex.ac.uk/chimera
  • 17. Multivariate analysis• Logistic approach – P(x) at t = control variables/known effects + treatments – Where X is • Refusal at t (responding hh) • Non-contact at t (responding hh) • Non-contact at t (non-responding hh) • Attrition www.essex.ac.uk/chimera
  • 18. Known effects• Based on Lynn et al (2005) Refuse Non-contact In HoL? Age Elderly Elderly & Young Y Income Lower Higher and/or employed Y Gender Men Y Education Less Y Composition singles singles Y Culture Ethnic minorities Y Mobility High mobility High mobility N Location Urban Urban N• In addition: – Technophobia (‘resonance’) – MRS code (AB, C1, C2, D,E) as proxy for wealth www.essex.ac.uk/chimera
  • 19. W2 resultsVariable w2 refusal w2 non-contact w2 non-contact (ind) (hh)Age -0.017* -0.062** -0.043***MRS Code 0.257** 0.158 0.225**Gender (female) -0.582** -1.428* -0.258*Qualification 0.095 -0.208 -0.048levelSingle person -0.821 0.399Ethnic minority -0.505 0.125Technophobia 0.028 0.164 0.049Call records 0.079 -0.735 -0.351Qualitative -0.875 1.831 -0.793Internet logging -0.678 -1.163Constant -1.961** -1.527 -0.565Pseudo r sq 0.044 0.128 0.08Chi sq 38.56854 20.49012 70.00324N 1172 881 1243 • logit, cluster (household identifier) [stata], values = b www.essex.ac.uk/chimera
  • 20. W3 results w3 w3 non-contact w3 non-contact refusal (ind) (hh)Age -0.02 -0.034 -0.026**MRS Code -0.06 -0.187 -0.181Gender -0.431 0.215 -0.149(female)Qualification 0.052 0.14 -0.016levelSingle person -0.175 0.206Ethnic minority 0.429 1.810** 0.459Technophobia 0.032 -0.162 -0.039Call records 0.445 -0.39 -0.001Qualitative -0.125 0.782 0.936Internet logging -0.243 1.234 0.326Constant -1.737* -0.778 0.298Pseudo r sq 0.022 0.106 0.035Chi sq 12.18588 34.6003 19.14218N 880 747 932• logit, cluster (household identifier) [stata], values = b www.essex.ac.uk/chimera
  • 21. Comparison• Statistically significant results only• logit, cluster (household identifier) [stata] www.essex.ac.uk/chimera
  • 22. Attrition Variable b Age -0.052*** MRS Code 0.175* Gender (female) -0.446*** Qualification level 0.004 Single person 0.005 Ethnic minority -0.122 Call records -0.313 Qualitative 0.055 Internet logging -1.566 Technophobia 0.064* Region (North) yorkshire & humberside 1.310** east midlands 0.357 east anglia 0.866 south east (excl. london) 1.089* south west 0.893 • Added region west midlands 0.516 north west 0.713 wales 0.802 scotland 0.656 • Call records variable ‘nearly’ greater london 1.322** significant (p = 0.066) Constant -0.403 Pseudo r sq 0.12 Chi sq 110.2714 N 1190• logit, cluster (household identifier) [stata] www.essex.ac.uk/chimera
  • 23. Conclusions I• Multi-method projects give you ‘better’ data• And ‘better’ results (see elsewhere)• But • They are resource hungry (researcher and respondent time/load) • They are complex to manage and analyse • You have to be multi-disciplinary/multi-skilled • All the usual qual/quant bickering takes place• All of which are good reasons to do them www.essex.ac.uk/chimera
  • 24. Conclusions II• Disappointingly: • Qualitative interviews did not help prevent non-response or attrition• BUT encouragingly • None of the ‘treatments’ were associated with non- response or attrition• So overall we should do this more often! www.essex.ac.uk/chimera
  • 25. Get the data• All 3 waves of the survey • UK Data Archive SN = 4607 • Free to UK Data Archive subscribers for non-commercial research• Held at Chimera (may be in UKDA eventually): • Qualitative transcripts • Call records (disclosure issues) • Internet usage logs www.essex.ac.uk/chimera
  • 26. Thank you• benander@essex.ac.uk www.essex.ac.uk/chimera

×