How and when does the mode of data collection affect survey measurement?

1. How and when does
the mode of data collection
affect survey measurement?
Annette Jäckle (University of Essex)
Pamela Campanelli (Independent Survey Methods Consultant)
Peter Lynn (University of Essex)
Gerry Nicolaas (National Center for Social Research)
Steven Hope (University College London)
Alita Nandi (University of Essex)
RSS 11 Oct 2011

2. Background
Existing guidelines for designing
questionnaires for use in different modes
US Census Bureau 2007
Dillman 2000

3. E.g. Census Bureau guidelines (2007)
Aim
meaning and intent of the question and
response options must be consistent across
modes.
Some differences
between modes in question wording may be
necessary to collect equivalent information
But
differences should be minimized and tested

4. Census Bureau guidelines (2007) ctd.
30 specific guidelines regarding
Q wording and instructions
Examples
Response categories
Formatting of answer spaces
Visual design elements
Question order and grouping
Showcards
Prompts and help

5. Census Bureau guidelines (2007) ctd.
Examples:
1. Maintain the same wording of questions
across modes.
6. The underlying response task posed by a
question should be consistent across modes
12. Use identical response categories
...

6. Census Bureau guidelines (2007) ctd.
But:
Even if Q wording, response options, task etc
are the same across modes
Mode effects sometimes occur
Why?
Which questions at risk?

7. Project aims
Effect of mode on measurement
Which survey Qs are at risk?
generalisable features of questions / modes
variable specific mechanisms
Practical advice for improving question
portability across modes
Funded by ESRC-SDMI

8. Today
Background
1. Framework, design of experiments
2. Cognitive interviewing follow-up study
Empirical results: the role of....
3. ...visual and aural stimuli
4. ...the interviewer
5. ...mode specific question formats
Implications
6. Questionnaire design for mixed mode surveys

9. Framework: causes of mode effects
Reporting situation
Perception of risk:
Extent of Interviewer Privacy of reporting
Legitimacy of survey Willingness to disclose?
involvement ► Social desirability bias
I-R interaction
Comprehension Retrieval Judgment Response
Depth of cognitive processing: Information available to R:
Sufficient effort? How is Q processed?
►Satisficing ►Response effects
Respondent Respondent motivation Task difficulty Context information
ability x Importance of survey x Interviewer characteristics
Reporting situation Cognitive Non-verbal communication
Non-verbal communication
•Time pressure demands Visual layout
Interviewer probing
•R distraction of Q Sequential/simultaneous Qs
Extent of Interviewer Respondent control
involvement Visual/aural stimulus Q format
over Qaire

10. Hypotheses: survey-level
Interviewers motivate respondents to make
required effort
⇒ Less satisficing
Interviewers can help with difficult tasks
⇒ Less confusion / fewer errors
Visual stimulus helps cognitive processing
⇒ Less satisficing / less confusion with visual
Format effects versus mode effects
⇒ No mode differences if same format

11. Hypotheses: item-level
Extent and nature of item-specific mode
effect depends on Q characteristics:
Question format
Inherent difficulty (content, wording)
Sensitivity
Type (attitude, behaviour, other factual, ...)
Level of measurement (ordinal, nominal, ...)
Response format (yes/no, frequency, ...)
Specific hypotheses in papers 3, 4, 5

12. Experimental design I
Split ballot question format experiments
Long vs. short list
Agree/disagree vs. forced choice
Ranking vs. rating
Full vs. end labels
Showcard vs. no showcard
Branching vs. no branching
Yes/no vs. code all that apply

13. Experimental design II
Each Q format experiment with several
items which varied by
Inherent Q difficulty (content, wording)
Sensitivity
Type (satisfaction, other attitude, behaviour,
other factual)
Measurement level (ordinal, nominal)

14. Experimental design III
Follow-up to NatCen Omnibus and BHPS
Great Britain population aged 16+
Restricted to respondents with web access
Random allocation to CAPI, CATI, CAWI

15. Response rates and sample sizes
Sample sizes after exclusion of non-internet access or use
cases from CAPI and CATI samples.
* Not available yet.

16. Today
Background
1. Framework, design of experiments
2. Cognitive interviewing follow-up study
Empirical results: the role of....
3. ...visual and aural stimuli
4. ...the interviewer
5. ...mode specific question formats
Implications
6. Questionnaire design for mixed mode surveys