Qualitative Researcg


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Qualitative Researcg

  1. 1. Survey Research Methods Week 3. Evaluating Surveys Robert Andersen, Steve Fisher and Anthony Heath Department of Sociology UNIVERSITY of OXFORD
  2. 2. Evaluation <ul><li>1. Results from previous surveys </li></ul><ul><li>2. Qualitative pre-testing/Questionnaire development </li></ul><ul><ul><li>Exploratory Interviews; Focus Groups; Expert Reviews; Cognitive Interviews </li></ul></ul><ul><li>3. Qualitative Field testing </li></ul><ul><ul><li>Respondent and interviewer debriefing </li></ul></ul><ul><li>4. Formal Quantitative tests </li></ul><ul><ul><li>Split panel tests; Analysis of item nonresponse and response distributions; behaviour coding </li></ul></ul>
  3. 3. 1) Exploratory interviews <ul><li>Individual one-on-one interviews </li></ul><ul><li>Largely informal, where respondents are able to ask questions as well </li></ul><ul><li>Mostly open-ended questions—Trying to determine appropriate response categories </li></ul><ul><li>Can be good for sensitive topics—Enables you to determine if the questions will even be answered at all </li></ul><ul><li>Often done several times—Do a few, revise questions, then do some more </li></ul>
  4. 4. 2) Focus Groups What are they? <ul><li>In-depth group interviews (5-8 people) run by a skilled moderator </li></ul><ul><li>Topic-based discussion </li></ul><ul><li>Purposively sampled participants </li></ul><ul><li>Usually meet on one occasion (usually taped) </li></ul><ul><li>What do focus groups tell us? </li></ul><ul><li>How people structure their thoughts about a topic </li></ul><ul><li>Understanding of general concepts </li></ul><ul><li>Use of terminology/vocabulary </li></ul><ul><li>Opinions about difficulty/sensitivity of questions </li></ul>
  5. 5. Focus Groups Issues and Advantages <ul><li>Before deciding on focus groups </li></ul><ul><ul><li>Is the topic suitable for group discussion? </li></ul></ul><ul><ul><li>Will group dynamics add value? </li></ul></ul><ul><ul><li>Remember: They won’t test the normal interview process </li></ul></ul><ul><li>Advantages of Focus Groups: </li></ul><ul><ul><li>Gather large amount of information quickly </li></ul></ul><ul><ul><li>Often raise issues that would not occur in one-to-one interviews </li></ul></ul><ul><ul><li>Can exploit group dynamics </li></ul></ul><ul><ul><li>Good at early stage when exploring a topic area </li></ul></ul><ul><ul><li>Discussing layout of forms, question wording at later stages </li></ul></ul>
  6. 6. 3) Expert/Interviewer Reviews Pre-Field Testing <ul><li>Formal and systematic scrutiny of a questionnaire by experts and/or experienced interviewers </li></ul><ul><li>Check list based on questionnaire design principles to identify potential problems </li></ul><ul><li>Can cover layout and routing as well as question wording </li></ul><ul><li>Won’t detect all problems but quick and cheap method prior to formal testing </li></ul><ul><li>Interviewers’ input especially good for CAI programs </li></ul>
  7. 7. 4) Cognitive interviews What are they? <ul><li>Individual interviews </li></ul><ul><li>Based on structured questionnaire— i.e ., questionnaire is fairly polished by this stage </li></ul><ul><li>Concerned with mental processes by which respondents reach answers </li></ul><ul><li>Usually audio or video recorded, perhaps for behaviour coding </li></ul><ul><li>Can be done in ‘laboratory’ (+ video, remote video, 2-way mirror etc.) </li></ul>
  8. 8. Cognitive Interviewing Techniques <ul><li>1. Probing questions </li></ul><ul><ul><li>Ask about uncertainties/difficulties </li></ul></ul><ul><ul><ul><li>‘ How did you arrive at your estimate of xxx?’ </li></ul></ul></ul><ul><ul><ul><li>‘ What did you understand by the word xxx?’ </li></ul></ul></ul><ul><ul><ul><li>‘ How confident are you in your answer?’ </li></ul></ul></ul><ul><li>2. Paraphrasing </li></ul><ul><ul><li>Respondents rephrase the question in their own words </li></ul></ul><ul><li>3. Concurrent ‘think-aloud’ </li></ul><ul><ul><li>Respondent vocalises thought processes as s/he answers </li></ul></ul><ul><li>4. Retrospective methods </li></ul><ul><ul><li>Probe and follow-up questions as interview progresses, or after the questionnaire is completed </li></ul></ul><ul><li>5. Behaviour Coding </li></ul><ul><ul><li>Identify deviations from the norm: Long pauses, questions of clarification, repetition, interviewer difficulties, etc. </li></ul></ul>
  9. 9. Cognitive Interviews Pros and Cons <ul><li>Concurrent methods or probes after each question simplify questions for respondent </li></ul><ul><li>However, concurrent methods can also disrupt the interview flow and relationship between questions </li></ul><ul><li>Interviews are quite long (1-1½ hrs) and therefore very costly, but don’t necessarily need many (perhaps up to 15) </li></ul><ul><li>Good for detailed wording testing </li></ul><ul><li>Doesn’t test under field conditions—perhaps unrealistic responses </li></ul>
  10. 10. Field Testing Interviewer and Respondent Debriefing <ul><li>Used to find out </li></ul><ul><ul><li>whether questions were understood </li></ul></ul><ul><ul><li>what respondent based answer on </li></ul></ul><ul><ul><li>ease of completing a task </li></ul></ul><ul><ul><li>which version of a question is preferable </li></ul></ul><ul><ul><li>whether questions missed issues </li></ul></ul><ul><li>Interviewer Debriefing </li></ul><ul><ul><li>Traditional method of evaluating questionnaires </li></ul></ul><ul><ul><li>Group discussions; Structured interviewer reports </li></ul></ul><ul><li>Respondent Debriefing </li></ul><ul><ul><li>Rating forms after pilot interviews or self-administered questionnaires </li></ul></ul>
  11. 11. Pros & Cons of field pre-tests/pilots <ul><li>Can combine non-response analysis, split sample tests with respondent and interviewer debriefing </li></ul><ul><li>Focus on the entire survey process, not just the questionnaire </li></ul><ul><li>Need to have a good idea of potential problems in advance </li></ul><ul><li>Take place late in development cycle limiting amount of change </li></ul><ul><li>Danger of making changes that are not tested afterwards, and thus possibly bringing in new sources of error </li></ul><ul><li>Testing takes time and money which are usually in short supply </li></ul>
  12. 12. Methods compared <ul><li>Experts: Very cheap and picks up more problems than any other single method </li></ul><ul><li>Interviewer debriefing: Costly but good at identifying interviewer problems </li></ul><ul><li>Cognitive interviewing: Low cost and good for identifying problems for data analysis </li></ul><ul><li>Behaviour coding: Costly but generally most consistent and reliable </li></ul><ul><li>Presser, S and Blair, J (1994) ‘Survey Pretesting: Do Different Methods Produce Different Results’, Sociological Methodology , 24:73-104 </li></ul>
  13. 13. Pre-testing Your Questionnaires <ul><li>Find subjects from other groups to pre-test your questions </li></ul><ul><li>Everyone should act as a subject and an interviewer for 15 minutes each </li></ul><ul><li>Experiment with different methods of testing </li></ul><ul><li>Regroup after half an hour to discuss the results of your pre-testing </li></ul>
  14. 14. Next week <ul><li>Presentation on sampling and survey administration </li></ul><ul><li>Further group work polishing your questionnaire </li></ul>