Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A best practices approach to cognitive interviewing

52 views

Published on

The Total Quality Framework (TQF) offers a conceptual foundation from which qualitative researchers are able to think about the quality of their cognitive interviewing designs. The TQF is rooted in the belief that cognitive interviewing and all qualitative research methods must be: credible, analyzable, transparent, and useful.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

A best practices approach to cognitive interviewing

  1. 1. A Best Practices Approach to Cognitive Interviewing Margaret R. Roller, MA Karen Kellard, MA July 18, 2019
  2. 2. July 18, 2019 1A Best PracticesApproach toCI – Roller & Kellard What is Qualitative Research?Overview  Qualitative research & cognitive interviewing  Quality in qualitative research & CI  Total Quality Framework (TQF)
  3. 3. July 18, 2019 2A Best PracticesApproach toCI – Roller & Kellard What is Qualitative Research?Qualitative Research & Cognitive Interviewing Attributes Absence of absolute “truth” Importance of context Importance of meaning Participant-researcher relationship Researcher skill set Flexibility of design Types of issues & questions Thematic analysis Researcher as instrument Unique online & mobile capabilities
  4. 4. July 18, 2019 3A Best PracticesApproach toCI – Roller & Kellard What is Qualitative Research?Qualitative Research & Cognitive Interviewing Attributes Absence of absolute “truth” Importance of context Importance of meaning Participant-researcher relationship Researcher skill set Flexibility of design Types of issues & questions Thematic analysis Researcher as instrument Unique online & mobile capabilities
  5. 5. July 18, 2019 4A Best PracticesApproach toCI – Roller & Kellard Qualitative Research & Cognitive Interviewing Challenges Importance of meaning Participant-researcher relationship Researcher skill set Flexibility Maximize Data Integrity
  6. 6. July 18, 2019 5A Best PracticesApproach toCI – Roller & Kellard  Strategies & criteria for thinking about quality in qualitative research are not new  Various post-hoc criteria for “trustworthiness,” validation & verification strategies, & guidelines (Brinkman & Kvale, 2015; Creswell, 2013; Levitt et al., 2017; Lincoln & Guba, 1985; Maxwell, 2013; Meyrick, 2006)  A quality approach specific to cognitive (CI) has also been recognized  “…cognitive interviewing practices sometimes lack appropriate scientific rigor…cognitive interviewers should strive for common standards…” (Willis, 2005, p. 134) Quality in Qualitative Research & CI
  7. 7. July 18, 2019 6A Best PracticesApproach toCI – Roller & Kellard Total Quality Framework  Develops critical thinking skills among researchers (& users of the research) by showing how to give explicit attention to quality issues at each stage of the research process regardless of the qualitative method “A useful tool…to (a) gather high-quality data, (b) lead to more robust and valid interpretations of the data, and (c) ultimately generate highly useful outcomes.” (Roller & Lavrakas, 2015, p.16)
  8. 8. July 18, 2019 7A Best PracticesApproach toCI – Roller & Kellard Total Quality Framework
  9. 9. July 18, 2019 8A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Credibility (Data Collection)
  10. 10. July 18, 2019 9A Best PracticesApproach toCI – Roller & Kellard SCOPE  Coverage • A cross-section of the population. • Inclusion of diverse geographic areas (as  Sample design • Purposive – not convenience or snowball – sampling. – Purposively select a subset of the population. • If working with targeted lists. – Stratify & select across the entire list.  Nonresponse • Gaining access to & cooperation from participants. TQF-Credibility (Data Collection)
  11. 11. July 18, 2019 10A Best PracticesApproach toCI – Roller & Kellard How Many Cognitive Interviews/Rounds to Complete? 1. Did every interview cover every critical question or area? What questions/areas were not adequately discussed with all participants? 4. Do the data obtained thus far effectively move the research to the next step? 7. Does a review of the observers’ & interviewers’ notes or reflexive journals reveal any concerns about objectivity and interpretations of the data?
  12. 12. July 18, 2019 11A Best PracticesApproach toCI – Roller & Kellard  Achieving valid outcomes  Content/information • Cognitive interview protocol.  Researcher effects • Interviewer bias. • Interviewer inconsistency.  Participant effects • Willingness/ability to provide information. DATA GATHERING TQF-Credibility (Data Collection)
  13. 13. July 18, 2019 12A Best PracticesApproach toCI – Roller & Kellard TQF-Credibility (Data Collection) Five Key Qualitative Interviewer Skills  Build rapport  Actively listen  Maintain sensitivity to verbal & nonverbal cues  Stay focused on the objectives, yet  Be flexible so as to not miss new learning
  14. 14. July 18, 2019 13A Best PracticesApproach toCI – Roller & Kellard TQF-Credibility (Data Collection) Probing Skills WHEN Concurrent Retrospective HOW Proactive Reactive WHAT Comprehension Recall Judgment Response
  15. 15. July 18, 2019 14A Best PracticesApproach toCI – Roller & Kellard  Evokes rationality. Researchers are in essence asking participants to justify their attitudes and behavior. In contemplating a justification, participants seek a response that “makes sense” or is otherwise deemed appropriate.  Confuses. “Why did this question make sense to you?” may be difficult and confusing to answer compared to “What are the specific aspects of of this question that helped you make sense of it?”.  Asks a different question from the one intended. The question “Why do you think this question is being asked?” is essentially a different question than “What do you think is the purpose of this question?” TQF-Credibility (Data Collection)
  16. 16. July 18, 2019 15A Best PracticesApproach toCI – Roller & Kellard TQF-Credibility (Data Collection)  Answers off-topic, answers a different question  Answers are too detailed, gives too much information  Answers are too short, too abrupt, lack detail  Fast talker, interviewer can’t keep up  Social desirability responding  Shy participant &/or uneasy with the CI environment  Uncomfortable with the think-aloud technique Types of Participants
  17. 17. July 18, 2019 16A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Analyzability (Analysis)
  18. 18. July 18, 2019 17A Best PracticesApproach toCI – Roller & Kellard PROCESSING  Transcriptions & coding • Are there resources for transcriptions & coding? • Is this the best approach?  Making sense of the data, consider: • Problematic areas – Identify & analyze the underlying cause for these problems » Question context – Impact of question(s) that came before • Researcher & participant effects – Consider how these effects potentially introduce error in the data • Vague or unclear responses – Importance of verification strategies TQF-Analyzability (Analysis)
  19. 19. July 18, 2019 18A Best PracticesApproach toCI – Roller & Kellard VERIFICATION  Add meaning & depth to the data to better inform researcher’s interpretations & recommendations  Evidence to support/refute findings  Types: • Peer debriefings • Triangulation (data, method, investigator) • Deviant cases (outliers, negative cases) • Reflexive journal (assumptions, beliefs, emotional connection, physical environment) TQF-Analyzability (Analysis)
  20. 20. July 18, 2019 19A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Transparency (Reporting)
  21. 21. July 18, 2019 20A Best PracticesApproach toCI – Roller & Kellard  “Thick description” of details in the final document  Give consumers of the research what they need to: • Understand the sample & the CI process. • Assign usefulness to the interpretations & recommendations. • Evaluate the transferability to other survey contexts. TQF-Transparency (Reporting)
  22. 22. July 18, 2019 21A Best PracticesApproach toCI – Roller & Kellard Thick Description Details for Cognitive Interviews
  23. 23. July 18, 2019 22A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Usefulness
  24. 24. July 18, 2019 23A Best PracticesApproach toCI – Roller & Kellard What is the TQF?TQF-Usefulness  Do the outcomes  Ultimately, reduce survey measurement error?  Confirm or deny what is known?  Enable researchers to transfer the learning to comparable contexts &/or types of participants?
  25. 25. July 18, 2019 24A Best PracticesApproach toCI – Roller & Kellard What is the TQF? The Total Quality Framework
  26. 26. July 18, 2019 25A Best PracticesApproach toCI – Roller & Kellard What is the TQF?  CI & qualitative research share unique attributes.  The important challenge is how to bring greater rigor to CI without stifling these unique attributes.  The Total Quality Framework (TQF) helps develop critical thinking skills re: best practices  Central to the TQF is the idea that our research must be credible, analyzable, transparent, & ultimately useful.  Not all best practices may be realistic for CI but the TQF offers a tool to give explicit attention to quality issues. Summary
  27. 27. Thank You! Please feel free to contact us Margaret R. Roller, MA Independent Consultant rmr@rollerresearch.com www.rollerresearch.com Karen Kellard, MA The Social Research Centre Melbourne AUSTRALIA karen.kellard@srcentre.com.au www.srcentre.com.au
  28. 28. July 18, 2019 27A Best PracticesApproach toCI – Roller & Kellard Reference: Text Roller & Lavrakas, 2015. New York: Guilford Press. http://bit.ly/TQFbook
  29. 29. July 18, 2019 28A Best PracticesApproach toCI – Roller & Kellard Reference: Blog http://www.researchdesignreview.com https://researchdesignreview.com/tag/total-quality- framework/
  30. 30. July 18, 2019 29A Best PracticesApproach toCI – Roller & Kellard Brinkmann, S., & Kvale, S. (2015). Interviews: Learning the craft of qualitative research interviewing (3rd ed.). Thousand Oaks, CA: Sage. Creswell, J. W. (2013). Qualitative inquiry & research design (3rd ed.). Thousand Oaks, CA: Sage Publications. Levitt, H. M., Motulsky, S. L., Wertz, F. J., Morrow, S. L., & Ponterotto, J. G. (2017). Recommendations for designing and reviewing qualitative research in psychology: Promoting methodological integrity. Qualitative Psychology, 4(1), 2–22. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications. Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Thousand Oaks, CA: Sage Publications. Meyrick, J. (2006). What is good qualitative research? A first step towards a comprehensive approach to judging rigour/quality. Journal of Health Psychology, 11(5), 799–808. Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press. Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications. ReferencesReferences

×