Downloaded 15 times
Uploaded byUniversity of Southampton
Cognitive interviewing
Cognitive interviewing is a qualitative research technique used to test survey questions by exploring respondents' thought processes as they answer. Techniques include think-aloud protocols, verbal probing, and observation, aiming to understand how participants comprehend and respond to questions. The process involves training respondents to verbalize their thoughts and is typically limited to 15-20 questions within an hour-long session.
More Related Content
More from University of Southampton
Cognitive interviewing
- 1.
- 2. What is cognitiveinterviewing?
- 3. Cognitive interviewing techniques •Think aloud • Probing • Observation • Response latency • Vignettes/ card sorts
- 4. Cognitive Interviewing Process Asksurvey Q Participant ‘thinks aloud’ Interviewer observation Verbal probing Comprehension Recall Judgment Response
- 6.
- 7. Training respondents tothink aloud • Explain format of the interview – Interviewer will ask a survey question/ ask respondent to attempt to fill in a questionnaire – Respondent is asked to verbalise thought processes • Practice thinking aloud – Interviewer demonstrates – Respondent has a go
- 8. Windows example • ADDANIMATION
- 9. How many windowsare there in your home? As you count the windows, tell me what you are thinking
- 10. Think aloud Usesnon-directive, open probing: ‘tell me more about what you’re thinking’ ‘keep talking’ ‘can you say more about that’
- 11.
- 12. Verbal probing Usesdirective, open questions what does the term X mean to you?’ ‘why did you choose that answer?’ ‘how did you remember that?’ ‘can you tell me, in your own words, what that question is asking?’ ‘Concurrent’ or ‘retrospective’
- 13. Concurrent and retrospectiveprobing • What do you think are the pros and cons?
- 14. Example of probing Testsurvey Q In the last two weeks, how many days have you spent outside?
- 15. Example of probing Exampleprobes • How did you go about answering this question? • What did you understand by the term outside when answering this question?
- 16.
- 17.
- 18. Number of questionsto test • Cognitive interview is usually no more than 1 hour • Need to allow time for – Introduction – Asking test survey questions – Think aloud – Probing • May only be able to test 15-20 survey questions
- 19. Number of testparticipants
- 20.
Editor's Notes
- #2 Hello. My name is Debbie Collins and I am going to introduce you to a method for pretesting your survey questions called cognitive interviewing. In this video I will describe what cognitive interviewing is and what it entails.
- #4 Cognitive interviewing is a set of qualitative techniques that can be used to explore people’s thought processes when presented with a task (e.g. Q) or information. Some of the techniques are listed on this slide Many of these rely on verbal reports – though some involve observation or measurement. E.g. response latency Think aloud and probing are the most widely used and we are going to focus on these two techniques. You can find about more about other techniques by looking at the resources.
- #5 So how does a cognitive interview work? This slide illustrates the process. This is a one-to-one interview The survey questions to be tested are administered in as close as possible a way to how they would be administered in the actual survey. They might be asked by an interviewer or presented on a mobile phone screen or tablet for the respondent to fill in. The respondent may be asked to think aloud as they attempt to answer the question The researcher may note the respondent’s behaviour, e.g did they express frustration or puzzlement? And either after each Q has been asked, a series or all test Q have been asked the respondent may be asked some further questions – verbal probes – about their thinking as they attempted to answer the survey Qs. Lets look a bit more at probing
- #11 As the name suggests, the purpose of Think Aloud is to encourage respondents to articulate their thoughts spontaneously as they are presented with a survey Q or task. At the start of the interview, respondents are trained in how to think aloud like this “Let me explain a little bit more about how the interview will work. I am going to ask you some questions. I want you to tell me whatever comes into your mind as you as you hear/read the question and try to answer it. This is called thinking aloud. We’ve found it helps to have some practice doing this, so let me give you an example. Let’s say I was asked “how many windows are there in my home?” If I was think aloud I would say … DEMONSTRATE Now let me ask you the same question: How many windows are there in your home? As you count the windows, tell me what you are thinking? During the interview the researcher encourages the respondent to think aloud, using non-directive, open probing such as those shown on the slide Not all Rs are willing or able to think aloud, an a criticism of the technique is that it favours the more articulate
- #13 Verbal probing used to explore specific aspects of the survey Q and Answer process (e.g. comprehension) Probes can be scripted and or spontaneous Examples of scripted probes on shown on the slide Spontaneous probes are used to follow up on respondents answers to the scripted probes, encouraging further expansion – can you tell me more about why that was confusing, or clarification. Concurrent or retrospective
- #15 This Q, asks about days spent outside in the past two weeks. It formed part of a series of Qs designed to measure sunlight exposure of the general population as part of a national survey. The number of days is recorded. Try to answer this question yourself. How easy or difficult did you find it to answer? We could ask respondents to think aloud as they try to answer this Q. Were you aware of what you were thinking as you tried to answer the Q? Here are a couple of cognitive probes that could be used to explore respondents’ thought processes when presented with it. What sorts of information might you get from these probes? Is there anything else you might want to explore? Have a go at writing some of your own probes to test your own survey questions, or write some for the survey questions provided in the Test Questions document contained in Resources. Have a go at doing a cognitive interview with a friend or colleague to test out your own survey questions or the test questions provided. Reflect on how you found the experience as the interviewer and find out how your respondent found it.
- #16 This Q, asks about days spent outside in the past two weeks. It formed part of a series of Qs designed to measure sunlight exposure of the general population as part of a national survey. The number of days is recorded. Try to answer this question yourself. How easy or difficult did you find it to answer? We could ask respondents to think aloud as they try to answer this Q. Were you aware of what you were thinking as you tried to answer the Q? Here are a couple of cognitive probes that could be used to explore respondents’ thought processes when presented with it. What sorts of information might you get from these probes? Is there anything else you might want to explore? Have a go at writing some of your own probes to test your own survey questions, or write some for the survey questions provided in the Test Questions document contained in Resources. Have a go at doing a cognitive interview with a friend or colleague to test out your own survey questions or the test questions provided. Reflect on how you found the experience as the interviewer and find out how your respondent found it.