Designing Questions forMixed Mode SurveysGerry Nicolaas National Centre for Social ResearchPamela Campanelli The Survey CoachAnnette Jäckle University of EssexPeter Lynn University of EssexSteven Hope University College LondonMargaret Blake National Centre for Social ResearchMichelle Gray National Centre for Social Research
Overview• Overview of question design strategies• Uni-mode construction• Practical advice• Ideas for further research• Conclusion
Mixing modes: question design strategies1. Uni-Mode Construction Writing and presenting questions the same to ensure respondents receive a common mental stimulus2. Mode-Specific Construction Modifying the question structure, wording or presentation for different modes based on the particular capabilities of each mode3. Mode-Enhancement Construction Using features not available in all modes to improve quality of responses in one particular mode4. Generalised Mode Design? Purposively designing questions differently in different modes with the aim of achieving cognitive equivalence References: Dillman et al (2009), de Leeuw (2005)
Mixing modes: question design strategies• Current situation Choice of strategies depends on combination of modes and question types/formats Identical wording and presentation is not always possible Minimal evidence on how to utilise mode-specific capabilities to achieve the same stimulus Minimal evidence on how to purposively design questions differently by mode to achieve cognitive equivalence• Main strategy used for this study Uni-mode construction
9 Uni-Mode Principles (Dillman 2000)1. Make all response options the same across modes and incorporate them into the stem of the survey question.2. Avoid inadvertently changing the basic question structure across modes in ways that change the stimulus. e.g. individual questions versus matrix, ‘yes/no’ versus ‘tick all that apply’3. Reduce the number of categories to achieve mode similarity.4. Use the same descriptive labels for response categories instead of depending upon people’s vision to convey the nature of a scale concept.
9 Uni-Mode Principles (Dillman 2000)5. If several items must be ranked, precede the ranking question with a rating question. rating acts as cognitive introduction to ranking task6. Develop equivalent instructions for skip patterns that are determined by answers to several widely separated items.7. Avoid question structures that unfold. because of increased skips in self-completion8. Reverse the order in which categories are listed in half the questionnaires.9. Evaluate interviewer instructions carefully for unintended response effects and consider their use for other modes.
Further research• Replication with many questions of each type• In addition to question difficulty & sensitivity, explore impact of other mediators/moderators• More experiments with general population samples• As well as question topic, need to look at other aspects of questionnaire design• Item-level theories about causes of mode effects on measurement• Experiments separating channels of communication of question stem and channels of communication of response options• Use of show cards in practice and how this relates to measurement• Use of validation studies in combination with mode studies
Conclusions (1)• “Mode” is more complex than simple distinction between face-to-face, telephone, postal, web Mode relates to a question not a survey A question can use more than one channel of communication Mode interacts with survey question format• Mixing modes increases risk of differences in measurement• Good question design is critical Cognitive processing and effort made to answer poorly designed questions will differ by mode, thus increasing the risk and magnitude of measurement differences
Conclusions (2)• Certain formats less likely to be portable across modes ‘Y/N for each’, branching, ‘agree/disagree’, end-labelled, ranking questions, rating questions• Equivalent question wording/task is not always sufficient• Uni-Mode Construction is likely to remain the dominant question strategy for mixed mode surveys in the near future• Cognitive interviewing can tell us something about mode effects Through gaining a greater understanding of how mode effects happen
Useful references• Overview of how to mix modes in surveys De Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21, 233–255. Retrieved from http://www.jos.nu Betts, P. And Lound, C. (2010). The Application of Alternative Modes of Data Collection in UK Government Social Surveys. Retrieved from http://www.ons.gov.uk/ons/about-ons/who-we- are/services/data-collection-methodology/reports-and- publications/alternative-modes-of-data-collection/index.html• Unimode Question Construction Dillman, D. (2000). Mail and Internet Surveys - The Tailored Design Method (2nd Edition), John Wiley and Sons: New York. Dillman D, Smyth J and Christian L (2009) Internet, Mail and Mixed-Mode Surveys - The Tailored Design Method (3rd Edition), John Wiley and Sons: New Jersey.
Useful references• Guidelines Martin E, Hunter Childs J, DeMaio T, Hill J, Reiser C, Gerber E, Styles K, Dillman D (2007) Guidelines for Designing Questionnaires for Administration in Different Modes, US Census Bureau, Washington. http://www.census.gov/srd/mode-guidelines.pdf
Thank you For further info about this study please contact Gerry.Nicolaas@natcen.ac.uk www.natcen.ac.uk