Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Questionnaire design


Published on

  • Be the first to comment

Questionnaire design

  1. 1. Topic 3POLS2110Jan 28 & Feb 4Questionnaire design
  2. 2. Questionnaire design: ProcedurePOLS2110-T31. Types of questions2. Question wording3. Formatting – Options4. As a whole: Cover letter, layout2
  3. 3. TypesPOLS2110-T31. Personal: Sex, age-group,…2. Attitudinal: attitudes or opinions “Agree” / “Support”3. Behavioral: E.g., Did you cast a vote during the LegislativeCouncil 2008?4. Closed- and open-ended questions Reasons given vs. you give reasons Will you offer “Others (reasons)”?3
  4. 4. TypesPOLS2110-T35. Sensitive Personal, social, cultural and political orientations:sexual orientation, drug & alcohol use, mentalhealth issues, illegal activities, controversial publicissues, political belief, identities. E.g., Do you agree that the June 4th Incident is a conspiracy? Do you take drug at school?4
  5. 5. Constructing questionnairePOLS2110-T3Guides for asking questions The surveyor should ask questions only fromthose likely to be able to answer them accurately Questions should be relevant: dont ask questionson topics that respondents dont care about, haventthought about, or that are irrelevant to the researchproblem Respondent must be competent to answer: dontask questions that the respondent wont accuratelybe able to answer5
  6. 6. Question wordingPOLS2110-T31. Specific questions/statements for specificanswers Avoid double-barreled questions (make sure thequestion asks only one clear thing) Each item should consist of one concept/term E.g., Do I care about my students and theirfamilies? How many concepts/terms? How would you revive the question?6
  7. 7. Question wordingPOLS2110-T32. Simple language: Clear and understandable avoid slang, jargon, and technical terms Short items are best (so that they may be read,understood, and answered quickly) E.g.,Do you agree that “the Legislative Council’s splitvoting system should remain in passing a bill”?你是否同意立法會在通過議案時繼續使用分組點票制度? (If you don‟t study HK politics), which term you find itdifficult to understand? How to change the question so as to make itunderstandable7
  8. 8. Question wordingPOLS2110-T33. Avoid ambiguous questions A word having at least two possible interpretations E.g.,Do you agree that “the public should supervisethe council”?你是否同意「在香港,議會應受到公眾的監督」? Which term seems to be ambiguous How to make changes?8
  9. 9. Question wordingPOLS2110-T34. Avoid vague words Make items clear: dont assume the person youare questioning knows the terms you are using When unique and unusual terms need to bedefined in questionnaire items, use very cleardefinitions9
  10. 10. Question wordingPOLS2110-T35. Avoid leading questions Avoid biased items and terms: be sensitive to theeffect of your wording on respondents. E.g.,Do you agree that “Rational voters should not cast a votefor those candidates who resist China and disturb HongKong”?你是否同意「理性的選民不應該在立法會選舉裡投反中亂港的候選人一票?」 What do you feel? Any possible implications? How to make changes?10
  11. 11. Question wordingPOLS2110-T3 Avoid negative or double negative items E.g.,- if you ask “whether librarians should not be paid more”,it will confuse respondents;- I am confident in teaching vs. I am not confident inteaching;- I like to teach well-behaved students vs. I do not like toteach badly-behaved student11
  12. 12. Question wordingPOLS2110-T36. Avoid presuming questions E.g., Do you agree with the view that “as June 4Turmoil is a conspiracy organized by the Westerncountries, the running of the candlelight vigilannually undermines the harmonious relationshipwith mainland China”?12
  13. 13. Question wordingPOLS2110-T37. Try to avoid hypothetical questions Except such hypothetical questions can tape therespondents‟ views from different perspectives Use and interpret cautiously E.g., If Hong Kong is no longer suitable for you,would you seek means to leave HK?1. Yes; 2. Would like to but can‟t; 3. NO; 4.DK/Refuse to answer (HKTP, 1999)13
  14. 14. Question wordingPOLS2110-T38. Decide carefully whether a question should beasked in a personalized form Use “I” as the subject “I support Henry Tang to be the next ChiefExecutive” VS“Henry Tang should be the next Chief Executive” Any differences? Which one would you prefer?14
  15. 15. Question wordingPOLS2110-T39. Avoid questions involving memory Factual accuracy is meaningless when knowing publicopinion (However,) In some cases, factual accuracy aims tointerpret the social/political meaning behind thechosen option E.g.,- For your understanding, July 1st is ___.A. a rally B. A public holiday C. a day that HKreturned to China- Can you name any three of the Legislative Councilmembers who are Civic Party‟s members?[Do you think this question is meaningful to evaluatethe level of understanding local politics fromrespondents?]15
  16. 16. Question wordingPOLS2110-T310. Deal with embarrassing, controversial andsensitive questions carefully E.g., Sexual, political orientations; marital status;age Possible solutions: Skip such questions? Put such questions at the end of the questionnaire?16
  17. 17. Question wordingPOLS2110-T3Ask the embarrassing questions in non-embarrassing ways. E.g.,“Did you kill your wife?”1. The Everybody approach: “As you know, many people have been killing their wives thesedays. Do you happen to have killed yours?”2. The numbered card: (face-to-face survey only) Would you please read off the number on this card whichcorresponds to what became of your wife?” (HAND CARD TORESPONDENT)1. Natural Death 2. I killed her 3. Other (What?)(GET CARD BACK FROM RESPONDENT BEFOREPROCEEDING!)17
  18. 18. Asking questions which requiresensitive answersPOLS2110-T3Respondents may distort their answers If the question poses an actual threat If they want to make themselves look good If they are maintaining their own self-image18
  19. 19. FormattingPOLS2110-T31. Multi-Point Rating e.g. Strongly Agree to Strongly Disagree Choose from 2-point rating scale up to 10-pointrating scale Specify rating labels Include/Exclude "Unable to Rate" option? Low to high or high to low rating label direction19
  20. 20. POLS2110-T320
  21. 21. POLS2110-T321
  22. 22. POLS2110-T322
  23. 23. RatingPOLS2110-T31. True scale: e.g., completely true, mostly true, somewhat true, slightly true,not at all true Completely true, generally true, generally untrue,completely untrue2. Always to never: e.g., Always, usually, sometimes, rarely, never3. Very good  very poor4. Important  not important at all/unimportant5. Likely  unlikely6. Smiley faces [^_^], [o_o], [-_-], [>_<]23
  24. 24. 2. Likert-scale/Semantic DifferentialPOLS2110-T3 5 rating points to 10 rating points Specific rating anchors Include/Exclude "Unable to Rate"24
  25. 25. POLS2110-T325
  26. 26. POLS2110-T326
  27. 27. POLS2110-T327
  28. 28. 3. Multiple choicePOLS2110-T3 Select one or select multiple Limit number of selections ("Select up to X choices") Randomize choices28
  29. 29. POLS2110-T329
  30. 30. 4. RankingPOLS2110-T3 Define the meaning of numbers (e.g., 1 is the mostfavorite/important whereas 5 is the least) Rank each of the given items without overlapping30
  31. 31. 5. open-ended questionsPOLS2110-T3 Comments, opinions, feelings, reasons, etc.31
  32. 32. POLS2110-T332
  33. 33. 6. Skip the irrelevant questionPOLS2110-T31. Do you watch TV news for local news?Yes ___ (Go to Q.2) No ___ (Go to Q.3)2. If yes, which channel do you watch for local news?A. TVB JadeB. TVB PearlC. ATV LocalD. ATV WorldE. Others: ____________________3. Do you watch for local news through other platforms?A. InternetB. NewspapersC. MagazinesD. Wisenews33
  34. 34. Recap: Common problemsPOLS2110-T31. Unclear question because of vague words2. Two or more questions3. Wordy or lengthy questions4. Question contains negatives5. Question contains jargon6. Response categories overlap7. Unbalanced response options8. Mismatch between the question & the responses9. Respondent does not have understanding to answerquestion10. Not all respondents can answer the question –branching needed34
  35. 35. SamplePOLS2110-T335
  36. 36. SamplePOLS2110-T336
  37. 37. SamplePOLS2110-T337
  38. 38. Cover letterPOLS2110-T3 What is the study is about, why it is important,how the study results will be used Why the respondent is important to the study How the respondent was selected Promise of confidentiality a photo number to call if the respondent hasquestions & who will be reached at that number Also, want good quality headed notepaper withsurvey organization (and sponsor included) andsignature by responsible researcher38
  39. 39. RecommendationsPOLS2110-T3 Keep it short. Think of 10p per word. How much$ can you save? Make sure purpose is unbiased: Poor e.g.,“It is important to keep big business from harmingthe environment, so we are doing this survey to getyour honest opinions on the extent to which you feelthe environment should be protected” Better e.g.,“It is unclear whether people want either more orless to be done by state government to protect theenvironment than is now being done…”39
  40. 40. Cover Letter SamplePOLS2110-T340
  41. 41. Questionnaire as a WholePOLS2110-T31. clear, specific instructions on the questionnaireon how to mark answersFor postal, typically have „tick‟ boxes  need overall instructions: “Please tick the box thatbest describes you or write in the informationrequested.” Need prominent “question by question” instructions(e.g., „Tick one box only”, “write number in box”) Both sets of instructions also needed to websurveys41
  42. 42. E.g,POLS2110-T3Q.1 How long have you/your household been a tenantof this landlord?Tick one box only Under 1 year 1-5 years 6-10 years 11-15 years 16+ years Don‟t know/can‟t remember Q.2 How many people usually live here in total?Write number in box42
  43. 43. Questionnaire as a WholePOLS2110-T32. First questions should Should apply to everyone Be easy to answer Be interesting Be connected to what R understands as topic ofsurvey Be closed Not be sensitive! So where should you put your demographicvariables?43
  44. 44. Questionnaire as a WholePOLS2110-T33. Meaningful order It is useful for the respondent to sense the flow, ornatural progression, of the instrument4. Definition (be sure these are part of thequestion) In postal surveys, R rarely reads instructionsbooklets In web surveys where underlined words providedefinitions, Rs rarely use Remember to put definitions first in interviewsurveys44
  45. 45. Questionnaire as a WholePOLS2110-T35. Routing If paper self-completion, avoid complex routing confuse or be ignored Usually best to have everyone answer everyquestion (i.e. no routing) If routing, use arrows as well as prominent explicittext45
  46. 46. E.g.,POLS2110-T3Q.1 Normally, do you work out everyday? No (Skip to 4) YesQ.2 (If yes) How many minutes per day to youwork out?_____ minutes per day…Q.446
  47. 47. Questionnaire as a WholePOLS2110-T36. Open-ended questions In self-completion, be sure to provide the appropriateamount of write-in-space (too much or too little can leadto item nonresponse)7. Sections In interview surveys, have the interview read out asentence (e.g., “The next few questions are about…”) In self-completion, use short one or two world labels thatare eye catching8. Question numbers In self-completion, start numbering with 1 and numberconsecutively. Avoid sub-parts even though you thinkthis will make the questionnaire look shorter.47
  48. 48. Questionnaire as a WholePOLS2110-T39. Do not cram lots of questions together with smallprint to try and save space Better to be generous with space to make the task lookeasy and to give R a sense of progress in filling in thequestionnaire.10. Place sensitive questions toward the end (but notthe very last questions!)11. “Thank you” Make sure interviewers thank the respondent In self-completion, write a large “thank you” at theend. Provide written instructions (at the end) about how toreturn the questionnaire If posted, supply a return envelope48