How and when does        the mode of data collection       affect survey measurement?           Annette Jäckle (University...
Background Existing guidelines for designing questionnaires for use in different modes   US Census Bureau 2007   Dillman 2...
E.g. Census Bureau guidelines (2007) Aim   meaning and intent of the question and   response options must be consistent ac...
Census Bureau guidelines (2007) ctd. 30 specific guidelines regarding   Q wording and instructions   Examples   Response c...
Census Bureau guidelines (2007) ctd. Examples: 1.  Maintain the same wording of questions     across modes. 6. The underly...
Census Bureau guidelines (2007) ctd. But:   Even if Q wording, response options, task etc   are the same across modes   Mo...
Project aims Effect of mode on measurement Which survey Qs are at risk?   generalisable features of questions / modes   va...
TodayBackground  1.   Framework, design of experiments  2.   Cognitive interviewing follow-up studyEmpirical results: the ...
Framework: causes of mode effects                                                   Reporting situation                   ...
Hypotheses: survey-level Interviewers motivate respondents to make required effort ⇒ Less   satisficing Interviewers can h...
Hypotheses: item-level Extent and nature of item-specific mode effect depends on Q characteristics:   Question format   In...
Experimental design I Split ballot question format experiments   Long vs. short list   Agree/disagree vs. forced choice   ...
Experimental design II Each Q format experiment with several items which varied by   Inherent Q difficulty (content, wordi...
Experimental design III Follow-up to NatCen Omnibus and BHPS   Great Britain population aged 16+   Restricted to responden...
Response rates and sample sizes  Sample sizes after exclusion of non-internet access or use  cases from CAPI and CATI samp...
TodayBackground  1.   Framework, design of experiments  2.   Cognitive interviewing follow-up studyEmpirical results: the ...
Upcoming SlideShare
Loading in …5
×

Rss Oct 2011 Mixed Modes Pres1

250 views

Published on

How and when does
the mode of data collection affect survey measurement?

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
250
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Rss Oct 2011 Mixed Modes Pres1

  1. 1. How and when does the mode of data collection affect survey measurement? Annette Jäckle (University of Essex)Pamela Campanelli (Independent Survey Methods Consultant) Peter Lynn (University of Essex) Gerry Nicolaas (National Center for Social Research) Steven Hope (University College London) Alita Nandi (University of Essex) RSS 11 Oct 2011
  2. 2. Background Existing guidelines for designing questionnaires for use in different modes US Census Bureau 2007 Dillman 2000
  3. 3. E.g. Census Bureau guidelines (2007) Aim meaning and intent of the question and response options must be consistent across modes. Some differences between modes in question wording may be necessary to collect equivalent information But differences should be minimized and tested
  4. 4. Census Bureau guidelines (2007) ctd. 30 specific guidelines regarding Q wording and instructions Examples Response categories Formatting of answer spaces Visual design elements Question order and grouping Showcards Prompts and help
  5. 5. Census Bureau guidelines (2007) ctd. Examples: 1. Maintain the same wording of questions across modes. 6. The underlying response task posed by a question should be consistent across modes 12. Use identical response categories ...
  6. 6. Census Bureau guidelines (2007) ctd. But: Even if Q wording, response options, task etc are the same across modes Mode effects sometimes occur Why? Which questions at risk?
  7. 7. Project aims Effect of mode on measurement Which survey Qs are at risk? generalisable features of questions / modes variable specific mechanisms Practical advice for improving question portability across modes Funded by ESRC-SDMI
  8. 8. TodayBackground 1. Framework, design of experiments 2. Cognitive interviewing follow-up studyEmpirical results: the role of.... 3. ...visual and aural stimuli 4. ...the interviewer 5. ...mode specific question formatsImplications 6. Questionnaire design for mixed mode surveys
  9. 9. Framework: causes of mode effects Reporting situation Perception of risk: Extent of Interviewer Privacy of reporting Legitimacy of survey Willingness to disclose? involvement ► Social desirability bias I-R interaction Comprehension Retrieval Judgment Response Depth of cognitive processing: Information available to R: Sufficient effort? How is Q processed? ►Satisficing ►Response effectsRespondent Respondent motivation Task difficulty Context informationability x Importance of survey x Interviewer characteristics Reporting situation Cognitive Non-verbal communication Non-verbal communication •Time pressure demands Visual layout Interviewer probing •R distraction of Q Sequential/simultaneous Qs Extent of Interviewer Respondent control involvement Visual/aural stimulus Q format over Qaire
  10. 10. Hypotheses: survey-level Interviewers motivate respondents to make required effort ⇒ Less satisficing Interviewers can help with difficult tasks ⇒ Less confusion / fewer errors Visual stimulus helps cognitive processing ⇒ Less satisficing / less confusion with visual Format effects versus mode effects ⇒ No mode differences if same format
  11. 11. Hypotheses: item-level Extent and nature of item-specific mode effect depends on Q characteristics: Question format Inherent difficulty (content, wording) Sensitivity Type (attitude, behaviour, other factual, ...) Level of measurement (ordinal, nominal, ...) Response format (yes/no, frequency, ...) Specific hypotheses in papers 3, 4, 5
  12. 12. Experimental design I Split ballot question format experiments Long vs. short list Agree/disagree vs. forced choice Ranking vs. rating Full vs. end labels Showcard vs. no showcard Branching vs. no branching Yes/no vs. code all that apply
  13. 13. Experimental design II Each Q format experiment with several items which varied by Inherent Q difficulty (content, wording) Sensitivity Type (satisfaction, other attitude, behaviour, other factual) Measurement level (ordinal, nominal)
  14. 14. Experimental design III Follow-up to NatCen Omnibus and BHPS Great Britain population aged 16+ Restricted to respondents with web access Random allocation to CAPI, CATI, CAWI
  15. 15. Response rates and sample sizes Sample sizes after exclusion of non-internet access or use cases from CAPI and CATI samples. * Not available yet.
  16. 16. TodayBackground 1. Framework, design of experiments 2. Cognitive interviewing follow-up studyEmpirical results: the role of.... 3. ...visual and aural stimuli 4. ...the interviewer 5. ...mode specific question formatsImplications 6. Questionnaire design for mixed mode surveys

×