The use of cognitive interviewingmethods to evaluate mode effects in                  survey questions                    ...
Acknowledgements• Members of the wider research team   • Gerry Nicolaas (NatCen)   • Steven Hope (University College Londo...
Outline of presentation•   Background and the wider research project•   Aims of the cognitive interviewing stage•   Novel ...
Aims of the CI phase• Used as a follow up methodology• To explore:  • how mode effects happen  • how respondents process q...
Novel use of CI?•   Used in the context of a methodological project•   As a follow up methodology (after a survey)•   Use ...
Quantitative sample• Came from a sample of survey participants (2008)• Those who agreed to give their email addresses were...
Thinking behind sample designfor CI•   Importance of reflecting response behaviours in    different modes•   Ideal would b...
Sample design for CI•   Regression analysis suggested an over    representation of Rs with lower socio-economic    status ...
Quotas for CI                CI sample                   (x37) Acquiescers     Rankers    Non-satisficers    (x18)        ...
Format of the cognitive interview• Three parts:  1. Interviewers read a one page introduction to     the study  2. Survey ...
Survey administration1. Questions administered in CAPI   •   Interviewer asking questions   •   Use of a paper questionnai...
Mode mimicking in CAPI
Mode mimicking in CATI
Mode mimicking in CAWI
Retrospective think-aloud (1)• Face-to-face• Interviewers read out the following script to each R:  “Now we come to the pa...
Retrospective think-aloud (2)• Interviewers reminded respondent of:   • The question they had answered   • The mode it had...
Retrospective think-aloud (3)•   Interviewer reminded respondent of any behaviour which could    signify a problem    • He...
CI techniques• Use of pre-scripted probes• Lack of interviewer flexibility with probing• Less of the exploratory probing• ...
Data management & analysis• Cognitive interview data audio recorded and  transcribed word for word• Data entered into qual...
Reflections & limitations• How effective was mode mimicking during the interviews?• How effective is ‘think aloud’ at capt...
Conclusions• Cognitive interviewing worked well as a method for  uncovering response processes in different modes• Explore...
Cognitive Interview Findings onMode Preferences
Mode Preferences In mode preference studies, it is typical for the mode the respondent experiences to be the one that the ...
Cognitive Interview QuestionsAt the end of the cognitive interview, interviewers asked   • “Overall what was it like to an...
Mode preference findings (1)•   CATI was the least preferred mode.•   No Rs picked it as their first chose (and only a few...
Mode preference findings (2)•   Preferences for CAPI versus CAWI were mixed. This relates to . . .•   R’s level of compute...
Findings from the Cognitive Interviewing     How they fit in with the rest of the analysis                 and presentations
Identifying mode effects• Identified respondents who showed ‘suspicious’  response patterns• Compared them with the remain...
Examples of findings (1)• Where quantitative methods suggested there was a  problem (e.g. acquiescence bias) but a qualita...
Examples of Findings (2)
Where cognitive findings fit in• Limited to investigating a subset of quantitative  results• Will be interspersed through ...
Rss Oct 2011 Mixed Modes Pres2
Upcoming SlideShare
Loading in …5
×

Rss Oct 2011 Mixed Modes Pres2

314 views

Published on

The use of cognitive interviewing methods to evaluate mode effects in survey questions

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
314
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Rss Oct 2011 Mixed Modes Pres2

  1. 1. The use of cognitive interviewingmethods to evaluate mode effects in survey questions Michelle Gray Margaret Blake Pamela Campanelli
  2. 2. Acknowledgements• Members of the wider research team • Gerry Nicolaas (NatCen) • Steven Hope (University College London, Institute of Child Health) • Peter Lynn (Institute for Social & Economic Research, University of Essex) • Anette Jäckle (Institute for Social & Economic Research, University of Essex) • Alita Nandi (Institute for Social & Economic Research, University of Essex)
  3. 3. Outline of presentation• Background and the wider research project• Aims of the cognitive interviewing stage• Novel implementation of CI method• Alterations in the CI techniques• Limitations of the methodology• Conclusions• Mode preferences• How cognitive interviewing findings fit in with the rest of the analysis
  4. 4. Aims of the CI phase• Used as a follow up methodology• To explore: • how mode effects happen • how respondents process questions in different modes • quantitative findings which relate to mode effects (where a pattern was observed)
  5. 5. Novel use of CI?• Used in the context of a methodological project• As a follow up methodology (after a survey)• Use of multiple modes in one interview• Sample selected based on satisficing behaviour• Goal was to provide insights into apparent mode effects
  6. 6. Quantitative sample• Came from a sample of survey participants (2008)• Those who agreed to give their email addresses were • Randomly assigned to one of three experimental groups • Surveyed (as part of the experiment) in CAPI, CATI or CAWI • Questionnaire comprised of a mix of original 2008 survey Qs + new Qs which had been purposely designed to test research team’s hypotheses• Initial quantitative analysis of results showed • Some hypotheses were supported • Whilst others were not (with some quite surprising findings!)
  7. 7. Thinking behind sample designfor CI• Importance of reflecting response behaviours in different modes• Ideal would be to select (and then compare in CI) • Respondents who answer Qs in different modes inconsistently • Respondents who show no mode effects• However…mode effects detected at aggregate level• But there were two response behaviours which could be detected at individual level – and when aggregated varied across mode 1. Agreeing to opposite statements Satisficing 2. Non-differentiation on a ranking task
  8. 8. Sample design for CI• Regression analysis suggested an over representation of Rs with lower socio-economic status and from minority ethnic groups amongst those who satisficed• Three quota groups were decided upon 1. “Acquiescers” (those who agreed to opposite statements) 2. “Rankers” (those who misunderstood the ranking tasks) 3. “Non-satisficers” (those who had not acquiesced & not made ranking errors and had opposite demographic & socio economic profile to those who had satisficed)
  9. 9. Quotas for CI CI sample (x37) Acquiescers Rankers Non-satisficers (x18) (x10) (x9)
  10. 10. Format of the cognitive interview• Three parts: 1. Interviewers read a one page introduction to the study 2. Survey questions were administered in CAPI, CATI and then CAWI 3. Interviewers carried out the ‘retrospective think-aloud’• Whole of the interview audio recorded (and later transcribed)
  11. 11. Survey administration1. Questions administered in CAPI • Interviewer asking questions • Use of a paper questionnaire (to record responses) • Use of show cards (where necessary)2. Next questions administered in CATI • Respondents & interviewers in separate rooms within the house • Use of paper questionnaire (to record responses) • Mobile (I) to landline (R) or mobile (I) to mobile (R) • Loudspeaker function on interviewer’s mobile phone3. Finally questions administered in CAWI • Respondent on interviewer’s laptop • Offline version • Privacy replicated
  12. 12. Mode mimicking in CAPI
  13. 13. Mode mimicking in CATI
  14. 14. Mode mimicking in CAWI
  15. 15. Retrospective think-aloud (1)• Face-to-face• Interviewers read out the following script to each R: “Now we come to the part where I will ask you some questions about the Topical Issues questions you just answered. Mostly, I will be asking you to tell me what came to your mind while you were answering the Topical Issues questions. This is called ‘thinking out loud’. We’ve found that it helps to have some practice at doing this. So let’s try an exercise which has nothing to do with the Topical Issues questions. For fun, I’d like you to: Think about how many windows there are in your house. As you count up the windows, tell me what you are seeing and thinking about. Well done. That’s what it is like to think out loud.”
  16. 16. Retrospective think-aloud (2)• Interviewers reminded respondent of: • The question they had answered • The mode it had been asked (in person, on the phone, on laptop) • Response the respondent had provided• Interviewers read out the following script to each R:“How we will proceed is that I will first remind you of a particular question or set questions you have just answered and I will remind you of your answers and then I will ask you to think out loud and tell me how you came up with your answers. While we are doing this, if you remember a specific question as being particularly easy or difficult or sensitive, it would help if you could comment on this. Also if there are specific places where you felt that the way the question was delivered (that is, with me sitting here with you, over the phone, and on the computer) made answering the question easier or more difficult, it would be helpful if you could comment on this.”
  17. 17. Retrospective think-aloud (3)• Interviewer reminded respondent of any behaviour which could signify a problem • Hesitancies • Laughs • Additional comments• Then, where appropriate, interviewer asked respondent a number of pre-scripted open probes • “Tell me your thinking behind choosing that category”• Where there had been response behaviour displayed (such as choosing same category in ranking tasks) asked other probes • “I noticed you picked the same category (twice / three times / on all 4 questions). If you had to choose, which would be the most important?
  18. 18. CI techniques• Use of pre-scripted probes• Lack of interviewer flexibility with probing• Less of the exploratory probing• For a few questions, asked the same survey question in a different way• Focus of probing was: • Not to assess performance of the questions • To gain insight into the thought processes of the respondent • Whether the mode of data collection impacted on the way they answered
  19. 19. Data management & analysis• Cognitive interview data audio recorded and transcribed word for word• Data entered into qualitative data management programme (‘Framework’)• Member of research team produced cognitive interview analysis plan • Built around set of original hypotheses • Analysis distributed between team • Analysis asked to look at findings which support hypotheses as well as surprising findings • Produce bullet point summaries on the areas for investigation
  20. 20. Reflections & limitations• How effective was mode mimicking during the interviews?• How effective is ‘think aloud’ at capturing respondents’ thought processes without impacting on their observed behaviour?• The quality of the questions included in the experiment• The style of probing used• The complexity of the task carried out by interviewers.• Technical difficulties (reliance on technology)• Analysis process (sometimes strayed into quantitative)• Confining the scope of analysis to mode effects
  21. 21. Conclusions• Cognitive interviewing worked well as a method for uncovering response processes in different modes• Explored quantitative findings in more detail• Mode mimicking (also) worked well • CI modes felt similar to real modes (according to Rs) • Probing is necessary in person• We would encourage the use of CI in novel ways such as • To look at how Qs perform in different modes • As a follow up methodology – after a quantitative survey (to shed light on results)
  22. 22. Cognitive Interview Findings onMode Preferences
  23. 23. Mode Preferences In mode preference studies, it is typical for the mode the respondent experiences to be the one that the respondent says he or she prefers. • For a household survey example see Groves and Kahn (1979) • For a school based survey example see Brener et al (2006) • For a business survey example see Tarnai and Paxson (2004) • For a longitudinal example see Smyth and Olson (2010)
  24. 24. Cognitive Interview QuestionsAt the end of the cognitive interview, interviewers asked • “Overall what was it like to answer the questionnaire in person versus over the phone versus by yourself on the computer?” • “What did you like or dislike about these 3 different ways you were asked the questions?”To our knowledge no previous study has administered multiple modes to the same respondents. • Should improve the validity of the mode preference data
  25. 25. Mode preference findings (1)• CATI was the least preferred mode.• No Rs picked it as their first chose (and only a few picked it as their second choice).• Reasons for Rs’ dislike of CATI • Harder to concentrate / can be distracted by other things • Feeling rushed / pressured on the phone • One gives less attention on the phone or has to work harder to think about the question and answer • Phone made the task harder because of the lack of visual stimulus and difficulty in hearing • A general dislike of the phone as a mode of communication
  26. 26. Mode preference findings (2)• Preferences for CAPI versus CAWI were mixed. This relates to . . .• R’s level of computer literacy• Valuing personal nature of face-to-face or privacy of CAWI• Controlling the pace of the CAWI interview, although some Rs felt CAWI made them go too quickly without thinking• CAPI leading to better quality data because. . . • One is more engaged (would “get more out of her”) • Answers more considered (more likely to answer and to give a genuine answer) • More committed to the interview and good answers
  27. 27. Findings from the Cognitive Interviewing How they fit in with the rest of the analysis and presentations
  28. 28. Identifying mode effects• Identified respondents who showed ‘suspicious’ response patterns• Compared them with the remaining respondents• Interrogated the data• Looking for explanations for these patterns• Did not go through each question and respondent in turn; analysis combined questions and respondents
  29. 29. Examples of findings (1)• Where quantitative methods suggested there was a problem (e.g. acquiescence bias) but a qualitative approach showed that respondents were able to justify their answers.• Where quantitative methods identified unexpected response patterns or potential mode effects but only a qualitative approach can provide an understanding of what is happening• Where quantitative methods do not identify any suspicious response patterns but a qualitative approach identifies potential problems.
  30. 30. Examples of Findings (2)
  31. 31. Where cognitive findings fit in• Limited to investigating a subset of quantitative results• Will be interspersed through out remaining presentations• May specifically address issue in quantitative presentation• May present a broader perspective• May present a narrower perspective (i.e., only a subset of questions or only a particular aspect to share)

×