Mixing modes of data collection is increasingly common in social surveys. It is often claimed that mixing modes has the potential of improving population coverage and reducing non-response bias, but it may come at the price of reduced data comparability. In this presentation, an overview of the causes of measurement differences is provided.

1. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
ESRC Research Methods Festival
8 July 2014, St Catherine’s College, Oxford
Understanding the causes of
measurement differences by mode
Gerry Nicolaas

2. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Contents
• Background
• What causes measurement to be different by mode?
• Question design strategies

3. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Background

4. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Different ways of mixing modes:
• Mixtures of means of communication
• Data collection mixtures, e.g.
• Different data items from the same person
• Same data items from different people
• Same data items from the same person at different time points
Source: Edith de Leeuw (2005)

5. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Focus of this presentation:
• Collecting the same data items using different modes
• Risk of differential measurement error because people may
answer questions differently depending on the mode
• Potential for reduced data comparability

6. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
What causes measurement to be
different by mode?

7. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Causes of mode effects (Jäckle et al, 2011)
Extent of Interviewer
involvement
Reporting situation
Privacy, legitimacy, interaction
Willingness to disclose?
► Social desirability bias
Comprehension Retrieval Judgment Response
Sufficient effort?
►Satisficing
Respondent
ability
Respondent
motivation
Task
difficulty
Context
information
Extent of Interviewer
involvement
Visual - Aural stimulus
Question
format
Respondent control
over questionnaire
How is question processed?
►Response effects

8. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
To sum up the causes of mode effects:
• Extent of interviewer involvement
• Visual versus aural stimulus
• Question format
• Respondent control over questionnaire

9. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Mode is more complex than simple distinction between:
• Face-to-face interview
• Telephone interview
• Mail questionnaire
• Online questionnaire

10. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
For example, face-to-face interview
No showcard With showcard With self-
administered
module
Audio-CASI
Question Aural Aural Visual Visual & Aural
recording
Response
options
Aural Visual Visual Visual & Aural
recording
Answer Oral Oral Written Typed

11. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Another example, a telephone interview
Traditional Response lists
mailed in
advance
Touchtone
Data Entry
Interactive
Voice
Recognition
Question Aural
(interviewer)
Aural
(interviewer)
Aural
(interviewer)
Aural (recording)
Response
options
Aural
(interviewer)
Visual Aural
(interviewer)
Aural (recording)
Answer Oral Oral Key Oral

12. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
A final example, an online questionnaire
Simple With audio With audio &
recording
With remote or
virtual
interviewer
Question Visual Visual & Aural Visual & Aural Visual & Aural
Response
options
Visual Visual & Aural Visual & Aural Visual & Aural
Answer Typed Typed Oral Oral

13. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Mode is a characteristic of
a question, not a survey.

14. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Question Design Strategies

15. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Question design strategies:
• Uni-Mode Construction
• Writing and presenting questions the same to ensure respondents
receive a common mental stimulus
• Mode-Specific Construction
• Modifying the question structure, wording or presentation for
different modes based on the particular capabilities of each mode
• Mode-Enhancement Construction
• Using features not available in all modes to improve quality of
responses in one particular mode – minimal evidence
• Generalised Mode Design
• Purposively designing questions differently in different modes with
the aim of achieving cognitive equivalence – minimal evidence
Source: Dillman et al (2009); de Leeuw (2005)

16. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Summing up:

17. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
To sum up:
• Causes of mode effects:
• Extent of interviewer involvement
• Visual versus aural stimulus
• Question format
• Respondent control over questionnaire
• Mode is a characteristic of a question, not a survey

18. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Thank you
Gerry.Nicolaas@ipsos.com | 020 xxxx xxxx 11/11/13
© Ipsos MORI This work was carried out in accordance with the requirements of the international quality standard for market research, ISO 20252:2006 and with the Ipsos MORI Terms and Conditions which can be found here

19. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
References
• Dillman, D., Smyth, J. & Christian, L.M. (2009). Internet, Mail and
Mixed-Mode Surveys: The Tailored Design Method, 3rd edition.
Hoboken, New Jersey: Wiley.
• Jäckle, A., Lynn, P., Campanelli, P., Nicolaas, G., & Hope, S.
(2011). How and When Does the Mode of Data Collection Affect
Survey Measurement? ESRA Conference, 21 July 2011.
• de Leeuw, E. (2005). To Mix or Not to Mix Data Collection
Modes in Surveys. Journal of Official Statistics, 21(2), 233-255.