The use of cognitive interviewing methods to evaluate mode effects in survey questions

1. The use of cognitive interviewing
methods to evaluate mode effects in
survey questions
Michelle Gray
Margaret Blake
Pamela Campanelli

2. Acknowledgements
• Members of the wider research team
• Gerry Nicolaas (NatCen)
• Steven Hope (University College London, Institute of Child
Health)
• Peter Lynn (Institute for Social & Economic Research,
University of Essex)
• Anette Jäckle (Institute for Social & Economic Research,
University of Essex)
• Alita Nandi (Institute for Social & Economic Research,
University of Essex)

3. Outline of presentation
• Background and the wider research project
• Aims of the cognitive interviewing stage
• Novel implementation of CI method
• Alterations in the CI techniques
• Limitations of the methodology
• Conclusions
• Mode preferences
• How cognitive interviewing findings fit in with
the rest of the analysis

4. Aims of the CI phase
• Used as a follow up methodology
• To explore:
• how mode effects happen
• how respondents process questions in different
modes
• quantitative findings which relate to mode effects
(where a pattern was observed)

5. Novel use of CI?
• Used in the context of a methodological project
• As a follow up methodology (after a survey)
• Use of multiple modes in one interview
• Sample selected based on satisficing behaviour
• Goal was to provide insights into apparent mode
effects

6. Quantitative sample
• Came from a sample of survey participants (2008)
• Those who agreed to give their email addresses were
• Randomly assigned to one of three experimental groups
• Surveyed (as part of the experiment) in CAPI, CATI or CAWI
• Questionnaire comprised of a mix of original 2008 survey Qs
+ new Qs which had been purposely designed to test
research team’s hypotheses
• Initial quantitative analysis of results showed
• Some hypotheses were supported
• Whilst others were not (with some quite surprising findings!)

7. Thinking behind sample design
for CI
• Importance of reflecting response behaviours in
different modes
• Ideal would be to select (and then compare in CI)
• Respondents who answer Qs in different modes
inconsistently
• Respondents who show no mode effects
• However…mode effects detected at aggregate level
• But there were two response behaviours which
could be detected at individual level – and when
aggregated varied across mode
1. Agreeing to opposite statements Satisficing
2. Non-differentiation on a ranking task

8. Sample design for CI
• Regression analysis suggested an over
representation of Rs with lower socio-economic
status and from minority ethnic groups amongst
those who satisficed
• Three quota groups were decided upon
1. “Acquiescers” (those who agreed to opposite statements)
2. “Rankers” (those who misunderstood the ranking tasks)
3. “Non-satisficers” (those who had not acquiesced & not
made ranking errors and had opposite demographic & socio
economic profile to those who had satisficed)

9. Quotas for CI
CI sample
(x37)
Acquiescers Rankers Non-satisficers
(x18) (x10) (x9)

10. Format of the cognitive interview
• Three parts:
1. Interviewers read a one page introduction to
the study
2. Survey questions were administered in CAPI,
CATI and then CAWI
3. Interviewers carried out the ‘retrospective
think-aloud’
• Whole of the interview audio recorded
(and later transcribed)

11. Survey administration
1. Questions administered in CAPI
• Interviewer asking questions
• Use of a paper questionnaire (to record responses)
• Use of show cards (where necessary)
2. Next questions administered in CATI
• Respondents & interviewers in separate rooms within the house
• Use of paper questionnaire (to record responses)
• Mobile (I) to landline (R) or mobile (I) to mobile (R)
• Loudspeaker function on interviewer’s mobile phone
3. Finally questions administered in CAWI
• Respondent on interviewer’s laptop
• Offline version
• Privacy replicated

12. Mode mimicking in CAPI

13. Mode mimicking in CATI

14. Mode mimicking in CAWI

15. Retrospective think-aloud (1)
• Face-to-face
• Interviewers read out the following script to each R:
“Now we come to the part where I will ask you some questions
about the Topical Issues questions you just answered. Mostly, I
will be asking you to tell me what came to your mind while you
were answering the Topical Issues questions. This is called
‘thinking out loud’. We’ve found that it helps to have some
practice at doing this. So let’s try an exercise which has nothing
to do with the Topical Issues questions. For fun, I’d like you to:
Think about how many windows there are in your house. As
you count up the windows, tell me what you are seeing and
thinking about. Well done. That’s what it is like to think out
loud.”

16. Retrospective think-aloud (2)
• Interviewers reminded respondent of:
• The question they had answered
• The mode it had been asked (in person, on the phone, on
laptop)
• Response the respondent had provided
• Interviewers read out the following script to each R:
“How we will proceed is that I will first remind you of a particular question
or set questions you have just answered and I will remind you of your
answers and then I will ask you to think out loud and tell me how you
came up with your answers. While we are doing this, if you remember
a specific question as being particularly easy or difficult or sensitive, it
would help if you could comment on this. Also if there are specific
places where you felt that the way the question was delivered (that is,
with me sitting here with you, over the phone, and on the computer)
made answering the question easier or more difficult, it would be
helpful if you could comment on this.”

17. Retrospective think-aloud (3)
• Interviewer reminded respondent of any behaviour which could
signify a problem
• Hesitancies
• Laughs
• Additional comments
• Then, where appropriate, interviewer asked respondent a
number of pre-scripted open probes
• “Tell me your thinking behind choosing that category”
• Where there had been response behaviour displayed (such as
choosing same category in ranking tasks) asked other probes
• “I noticed you picked the same category (twice / three times / on all
4 questions). If you had to choose, which would be the most
important?

18. CI techniques
• Use of pre-scripted probes
• Lack of interviewer flexibility with probing
• Less of the exploratory probing
• For a few questions, asked the same survey question
in a different way
• Focus of probing was:
• Not to assess performance of the questions
• To gain insight into the thought processes of the respondent
• Whether the mode of data collection impacted on the way
they answered

19. Data management & analysis
• Cognitive interview data audio recorded and
transcribed word for word
• Data entered into qualitative data management
programme (‘Framework’)
• Member of research team produced cognitive
interview analysis plan
• Built around set of original hypotheses
• Analysis distributed between team
• Analysis asked to look at findings which support hypotheses
as well as surprising findings
• Produce bullet point summaries on the areas for
investigation

20. Reflections & limitations
• How effective was mode mimicking during the interviews?
• How effective is ‘think aloud’ at capturing respondents’
thought processes without impacting on their observed
behaviour?
• The quality of the questions included in the experiment
• The style of probing used
• The complexity of the task carried out by interviewers.
• Technical difficulties (reliance on technology)
• Analysis process (sometimes strayed into quantitative)
• Confining the scope of analysis to mode effects

21. Conclusions
• Cognitive interviewing worked well as a method for
uncovering response processes in different modes
• Explored quantitative findings in more detail
• Mode mimicking (also) worked well
• CI modes felt similar to real modes (according to Rs)
• Probing is necessary in person
• We would encourage the use of CI in novel ways
such as
• To look at how Qs perform in different modes
• As a follow up methodology – after a quantitative survey (to
shed light on results)

22. Cognitive Interview Findings on
Mode Preferences

23. Mode Preferences
In mode preference studies, it is typical for the mode the
respondent experiences to be the one that the
respondent says he or she prefers.
• For a household survey example
see Groves and Kahn (1979)
• For a school based survey example
see Brener et al (2006)
• For a business survey example
see Tarnai and Paxson (2004)
• For a longitudinal example
see Smyth and Olson (2010)

24. Cognitive Interview Questions
At the end of the cognitive interview, interviewers asked
• “Overall what was it like to answer the questionnaire in person
versus over the phone versus by yourself on the computer?”
• “What did you like or dislike about these 3 different ways you
were asked the questions?”
To our knowledge no previous study has administered
multiple modes to the same respondents.
• Should improve the validity of the mode preference data

25. Mode preference findings (1)
• CATI was the least preferred mode.
• No Rs picked it as their first chose (and only a few
picked it as their second choice).
• Reasons for Rs’ dislike of CATI
• Harder to concentrate / can be distracted by other things
• Feeling rushed / pressured on the phone
• One gives less attention on the phone or has to work harder to think
about the question and answer
• Phone made the task harder because of the lack of visual stimulus and
difficulty in hearing
• A general dislike of the phone as a mode of communication

26. Mode preference findings (2)
• Preferences for CAPI versus CAWI were mixed. This relates to . . .
• R’s level of computer literacy
• Valuing personal nature of face-to-face or privacy of CAWI
• Controlling the pace of the CAWI interview, although some Rs felt
CAWI made them go too quickly without thinking
• CAPI leading to better quality data because. . .
• One is more engaged (would “get more out of her”)
• Answers more considered (more likely to answer and to give a genuine answer)
• More committed to the interview and good answers

27. Findings from the Cognitive Interviewing
How they fit in with the rest of the analysis
and presentations

28. Identifying mode effects
• Identified respondents who showed ‘suspicious’
response patterns
• Compared them with the remaining respondents
• Interrogated the data
• Looking for explanations for these patterns
• Did not go through each question and respondent in
turn; analysis combined questions and respondents

29. Examples of findings (1)
• Where quantitative methods suggested there was a
problem (e.g. acquiescence bias) but a qualitative
approach showed that respondents were able to
justify their answers.
• Where quantitative methods identified unexpected
response patterns or potential mode effects but only
a qualitative approach can provide an understanding
of what is happening
• Where quantitative methods do not identify any
suspicious response patterns but a qualitative
approach identifies potential problems.

30. Examples of Findings (2)

31. Where cognitive findings fit in
• Limited to investigating a subset of quantitative
results
• Will be interspersed through out remaining
presentations
• May specifically address issue in quantitative
presentation
• May present a broader perspective
• May present a narrower perspective (i.e., only a
subset of questions or only a particular aspect to
share)