NYAAPOR: Question Wording and Questionnaire Design


Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

NYAAPOR: Question Wording and Questionnaire Design

  1. 1. Gary LangerLanger Research Associatesglanger@langerresearch.comNYAAPOR 6/6/13
  2. 2. Fasten your…
  3. 3. The Seat Belt (courtesy of Floyd Fowler)When you sit in the back seat of a car, how often do youuse the seat belt – always, sometimes, rarely or never?The problem:A quarter of Americans never sit in the back seat of a car.
  4. 4. Their options:a) Tell you, uninvited, that they never sit in the back seat.b) Think of what they usually do in the front seat, figurethey’d also do that if they ever sat in the back seat (whichthey don’t), and tell you they guess they’d do that.c) Use the offered option that best approaches reality:“Never.” But meaning that they never sit in the back seat,not that they never use their seat belt when they do.Because they don’t.What they do: c)
  5. 5. The result:Vast overstatement of the non-use of back-seat seatbelts.
  6. 6. What this is about:a) Presumptionb) CognitionPresumption, meaning: We know what we want to find out, so, dammit, whydoesn’t everyone else?Cognition, meaning: How people think about, understand and answer ourquestions. And how we make sense of their answers.
  7. 7.  Have you ever seen anything that you thought was a UFO, ornot? Gallup 6/90Saw UFO: 14% Have you ever seen anything that you thought was a UFO, ornot? ABC 6/97Saw UFO: 13% (If yes:) Do you believe what you saw was a spacecraft fromanother planet, or do you think it was something man-madeor some natural phenomenon that you could not identify?Thought it was an alien spacecraft: 4%
  8. 8. The risks of non-specification We can project meaning where it’s unintended by therespondent. The respondent can project meaning where it’smisinterpreted by us.
  9. 9. Expressed (vs. affirmed) belief Before the war, do you think Iraq did or did notprovide direct support to the Al Qaeda terroristgroup? ABC/Post ’05Did: 61% If yes: (Is that your suspicion only), or (do youthink theres been solid evidence of that)?Evidence: 21%
  10. 10. Is President Obama… A Muslim?… Kenyan-born?…“Poll: 1 in 5 Americans Believe Obama is a Cactus”(The Onion, 2010)(See also climate change.)
  11. 11. Context, too Trust in government…to do what? Pre 9/11: Muck around with social policy Post 9/11: Keep us from getting killed by terrorists Post 2008: Screw up the economy
  12. 12. The solutionRespect this thing.
  13. 13. The best way to stay out of troubleDon’t ask foolish questions.(Yes. In polling, there are foolish questions.)
  14. 14. How we (try to) avoid them• Learn from the literature• Acquiescence bias• Agree/disagree, yes/no, true/false• Satisficing/cognitive burden• Positive attribute bias• Optimal vs. suboptimal response scales• Numbers/words, categories, labeling, branching The midpoint controversy (resolution may be purpose-driven)• Priming• Order effects/list effects• Good news soon, at a bookstore near you
  15. 15.  Mind the basics Balance Proportion Cover the options Open-ends can help find them Mutual exclusivity (e.g. walk/lunch) Alt. prop. Simplicity Clarity Brevity Intros can stack the cards. Some/other people can, too Avoid jargon, euphemisms and hot buttons Be aware of kvetchism and the danger of retroactive trend Know the limits of recallFurther foolishness avoidance
  16. 16. Know thy purpose and find a path Practice risk assessment: Where will this go wrong? Know who you’re talking to Know what you’re trying to find out Research what’s known, but skeptically Experiment Stack not the cards, load not the dice Recognize the limitations of any single question
  17. 17. Have a seat…
  18. 18.  Do you support or oppose the death penalty forconvicted murderers? (68%) What do you prefer for convicted murderers, the(death penalty) or (life in prison with no possibilityof parole)? (49%) Do you support or oppose the death penalty for……Timothy McVeigh (75%)…Murderers who are mentally retarded? (19%)…Murderers who were juveniles at the time of thecrime? (26/49/23%) (Note the vols!)
  19. 19. Same with abortion… and so many others…. Support or oppose legal abortion? (55%)…To protect woman’s life? (88%)…In cases of rape or incest (81%)…To terminate unwanted pregnancy? (42%)…After 1st trimester (11%)…Private matter? Murder?
  20. 20. Dare disagree? Do you agree or disagree with this statement: “Abortion is murder.”LA Times 3/89Agree: 58% Which of these statements comes closer to your opinion –abortionis the same thing as murdering a child, or abortion is not murderbecause the fetus really isnt a child? CBS/NYT 4/89Murder: 48% Some people say that abortion is an act of murder, while otherpeople disagree with this. What is your view – do you thinkabortion is an act of murder or dont you feel this way? (Time/CNN 8/94)Murder: 42%
  21. 21. Thanks, Willem and Jon“A huge body of research conducted during more than fivedecades has documented the role that acquiescence responsebias plays in distorting answers to agree/disagree questions.”“… remarkably sizable differences in data quality” inagree/disagree vs. balanced forced-choice formats.-Saris and Krosnick, “Comparing Questions with Agree/Disagree ResponseOptions to Questions with Construct-Specific Response Options.”
  22. 22. Why? Absence of the alternative proposition is inherentlyunbalanced; Common “solution,” rotating positive and negativestatements, merely shifts the bias rather than resolving it; Format encourages acquiescence; and Satisficing to minimize cognitive burden.
  23. 23. Why? For a headline, maybe“Significant numbers of African Americans believe inconspiracy theories about AIDS…” RAND Corp./Oregon StateUniversity, 1/05“More than 29 years after the AIDS virus arrived in theUnited States, a significant proportion of AfricanAmericans embrace the theory that governmentscientists created the disease to control or wipe outtheir communities, according to a study released todayby Rand Corp. and Oregon State University.” Washington Post,1/25/05
  24. 24. First, I’m going to read you several statements regarding HIV and AIDS. Foreach one, I’d like you to tell me whether you agree strongly, agree somewhat,have no opinion, disagree somewhat, or disagree strongly. The first one is...1.1 The medicines used to treat HIV are saving lives in the black community.1.2 A lot of information about AIDS is being held back from the public.1.3 HIV is a manmade virus.1.4 There is a cure for AIDS, but it is being withheld from the poor.1.5 The government is telling the truth about AIDS.1.6 The medicine used to treat HIV causes people to get AIDS.1.7 HIV was created and spread by the CIA.1.8 AIDS is a form of genocide against Blacks.1.9 The medicine that doctors prescribe to treat HIV is poison.1.10 AIDS was created by the government to control the Black population.1.11 Doctors put HIV into condoms.1.12 People who take the new medicines for HIV are human guinea pigs for the government.1.13 Medical and public health institutions are trying to stop the spread of HIV inblack communities.1.14 AIDS was produced in a government laboratory.
  25. 25. Further headaches… “Significant proportion” was 16 percent in agree/disagree. Sample, described as “a random sample of 500 African-Americans,”* was limited to high-density (>26%) blackCensus tracts; noncoverage 49.5%. Survey introduction twice says it’s “about discrimination” Sample limited to 15- to 44-year-olds.*Journal of Acquired Immune Deficiency Syndrome, 2/05
  26. 26.  With the economy the way it is, some women may havechanged their plans about when or how many children tohave. Is the following statement true for you? Because ofthe economy, I now want to have a baby later than I hadpreviously planned Guttmacher Institute, 9/09 (Sexually active women 18-34 <$75K)True: 44% As a result of the economic slump, have you postponed orcanceled any of the following? Having a baby Time 4/09, women18-34 <$75K (N=56)Yes: 18%Y/N, T/F, too
  27. 27. Priming the pump… As a result of the recent increase in gas prices have you...cutback on your driving in any way? Newsweek 9/05Driving less: 71% Thinking about how much driving you do, would you saythat compared to this time last year youre driving more,less, or about the same amount? (If less, ask:) Is thatbecause of the price of gasoline, or for some other reason?ABC/Post 8/05Driving less: 31% (Because of gas prices, 22%)
  28. 28. Did I say aagh?Stephen,We recently used a very similar question to that used by Paul for a dual-frame survey in Australia …“Research companies sometimes invite members of the general public tobecome members of online research panels where they receive points orincentives for regularly completing surveys online. Are you currently amember of a research panel?”Cheers,DarrenManaging Director and Head of Research…(AAPORNET, 6/5/13)
  29. 29. Let’s tryResearch companies sometimes invite members ofthe general public to become members of onlineresearch panels where they receive points orincentives for regularly completing surveys online.Are you currently a member of a research panel?How about… (just a starter)Do you ever take surveys over the internet in exchangefor cash or gifts, or not?
  30. 30. Let’s give this a go
  31. 31. Gay marriage Do you support or oppose gay marriage? Do you think it should be legal or illegal for gays andlesbians to get married? Do you favor or oppose allowing gays and lesbians tomarry legally? Consider further: homosexuals, gays and lesbians,same-sex couples, men/men and women/women?
  32. 32. The hall of horrors
  33. 33. What’s the dope? “As you may know, methamphetamine use has risendramatically in counties across the nation……Have arrests where methamphetamine was involvedincreased in your county?”...Which of the following drugs is the biggest problemin your county…?National Association of Counties, 6/05 (n.b., convenience sample)
  34. 34. “Knowledge” Polls Knowledge vs. recall Opinion vs. behavior vs. recall The operation and activation of memory The weight of error The effect of mode/method (No. 2 pencils) The attempt to humiliate
  35. 35. Just say no.13. Socrates, Plato, Aristotle, and Aquinas would concur that:a. all moral and political truth is relative to one’s time and placeb. moral ideas are best explained as material accidents or byproducts of evolutionc. values originating in one’s conscience cannot be judged by othersd. Christianity is the only true religion and should rule the statee. certain permanent moral and political truths are accessible to human reason27. Free markets typically secure more economic prosperity than government’s centralized planningbecause:a. the price system utilizes more local knowledge of means and endsb. markets rely upon coercion, whereas government relies upon voluntary compliance with the lawc. more tax revenue can be generated from free enterprised. property rights and contracts are best enforced by the market systeme. government planners are too cautious in spending taxpayers’ money29. A flood-control levee (or national defense) is considered a public good because:a. citizens value it as much as bread and medicineb. a resident can benefit from it without directly paying for itc. government construction contracts increase employmentd. insurance companies cannot afford to replace all houses after a floode. government pays for its construction, not citizensIntercollegiate Studies Institute 5/08
  36. 36. The Black Art:Subtle and unsubtle manipulations
  37. 37. Sounds good… “A recent survey shows that two-thirds of women would gladly reducethe number of monthly periods if they could.” (ABC PrimeTime 9/19/03) “If there was a safe and easy way, how interested would you be inreducing your period and related symptoms, such as PMS or bloating,to only four times a year or once every three months? Would you beextremely interested, very interested, somewhat interested, not veryinterested, or not at all interested in reducing your period to only fourtimes a year?”• Positive bias: “safe,” “easy,” “reduced symptoms.”• No mention of medication, cost, side effects.• 3-2 unbalanced• Produced in Barr Labs promotional campaign
  38. 38. Drive it away… Do you believe that the government should or should not provide loansto America’s automakers so they have the money to manufacturevehicles?Should Should not No opin.55% 30 15 The big three automakers in the United States have asked for a 25billion dollar loan from the government. Some people say (its a bailoutthose companies dont deserve, and that theyd be better offreorganizing under bankruptcy laws). Other people say (its necessaryto protect auto workers and save a key part of the U.S. economy). Onbalance, do you support or oppose this plan?Support Oppose No opin.35% 57 9
  39. 39. Behind the curtain…1. How important do you feel the American automobile industry is to theAmerican economy--extremely important, very important, somewhatimportant, not important, or not at all important?2. If the American automobile industry no longer had the resources toproduce vehicles, how much harm would it cause to... INSERT – a great dealof harm, quite a bit of harm, just some harm, or very little harm?a. Americas manufacturing job sectorb. the American economyc. Americas standing in the worldd. consumer choice for Americas car buyers3. Do you believe that the government should or should not provide loans toAmericas automakers so they have the money to manufacture vehicles?Hart Research for GM, 11/08
  40. 40. Let’s Dance. I’ll lead… Let me read you four facts about the auto industrys effect on the Americaneconomy.(1) The U.S. auto industry affects numerous other American industriesbecause it is the largest purchaser of steel, copper, plastics, electronics, andcomputer chips.(2) The U.S. auto industry provides pensions to seven hundred and seventy-five thousand Americans and health benefits to two million Americans.(3) The U.S. auto industry supports approximately five million U.S. jobs in allfifty states.The U.S. auto industry has fourteen thousand dealers in nearly every town inAmerica employing seven hundred thirty thousand people.Knowing this, if General Motors, Ford, and Chrysler were to go out ofbusiness in the next few months, how likely do you think this would be totrigger an economic depression in the U.S. – extremely likely, very likely,somewhat likely, not likely, or not at all likely? (3-2 = 79%)
  41. 41. Lowdown on the down-lowPublic release date: 18-Sep-2006American College of PhysiciansIn new survey, men call themselves straight but have sex with menPHILADELPHIA, September 19, 2006 – A survey of 4,193 men living in New York City conducted by the New York CityDepartment of Health and Mental Hygiene found that nearly 10 percent of male participants who identified themselves asstraight reported having sex with at least one man during the previous year.The study, "Discordance between Sexual Behavior and Self-Reported Sexual Identity: A Population-Based Survey of NewYork City Men," is published in the Sept. 19, 2006, issue of Annals of Internal Medicine.Compared to men who identified themselves as gay, these men were more likely to belong to a minority racial or ethnicgroup, be foreign-born, have a lower educational level, and live outside Manhattan. Seventy percent reported beingmarried. This group also was less likely to have been tested for HIV infection during the previous year and less likely to haveused a condom during the last sexual encounter than men who identified themselves as gay."Doctors need to ask patients about specific sexual practices instead of relying on self-reported sexual orientation to assessrisk for unsafe sexual practices and risk for sexually transmitted diseases," said Preeti Pathela, DrPH, lead author of thestudy. "Public health prevention messages should target risky sexual activities, such as unprotected receptive anal sex, andshould not be framed to appeal solely to gay-identified men."This study is one of the largest U.S. population-based surveys to report on the contrast between a mans self-identifiedsexual identity and his actual sexual behaviors.###NOTE TO EDITORS: To interview Preeti Pathela, DrPH or Harold Sox, MD, Editor of Annals of Internal Medicine, contactSteve Majewski at 215-351-2514 or smajewski@acponline.org.43
  42. 42. 44Limitations and Findings First hurdle: Sampling methodology is top drawer; study is partof federal BRFSS. “Limitations: The survey did not sample groups that cannot bereached by using residential telephone services.” “Conclusions: Many New York City men who have sex with mendo not identify as gay.”
  43. 43. Almost one in 10 straight menon the “down-low,” study finds9/18/2006By Dawn FallikThe Philadelphia InquirerAlmost 10 percent of men who say theyre straight alsohappen to be having sex with men, according to a new study,one of the largest ever to specifically address "down-low"behavior.The study, based out of New York City, found that most of thedown-low men did not use condoms and that 70 percentwere married. Researchers said they hoped their reportwould change the way doctors asked patients about theirsexual behavior.45
  44. 44. 46What they asked(Figure 1)“Question: ‘During the past 12 months, with how many men haveyou had sex?’“Interviewer read the following: ‘By sex we mean oral, vaginal, oranal sex, but not masturbation.’
  45. 45. 47Whoa. Last thing respondent heard was not about sexwith men, but about sex including vaginal sex. Self-identified heterosexual men who reportedhaving sex (including vaginal sex) with other menwere disproportionately foreign born, low-education, married (70%), apt to report a single sexpartner in the previous year (96%) and less likely touse a condom or be HIV-tested.
  46. 46. Did they misunderstand the question?Were they reporting sex…with their wives?48
  47. 47. 49In the 30th paragraph (of 35) “It is possible that this unique method, in whichmen were asked about same-sex partners first andthe accompanying definition of sex includedvaginal intercourse, resulted in misunderstandingof the question…” “Preliminary data from the 2005 CHS, whichascertained sex of partners by asking aboutopposite-sex partners first, suggest that doing soresulted in a much lower self-reported prevalenceof men who report sex with other men.(…unpublished data.)”
  48. 48.  Subsequent response from the California Department ofHealth Services reports a statewide estimate of 0.4 percentof straight men on the down-low, compared with the 9.4percent in the NYC study. The California researchers conclude that the NYC finding"is inconsistent with the rates reported by others, and mayin fact be an artifact of the methods used in the survey."http://www.annals.org/cgi/eletters/145/6/416#391850
  49. 49. What we want• No games• Neutral, balanced wording, w/ alternativeproposition.• Simple phrasings with clear meanings• Realistic and proportionate response options• Rotations where appropriate• Sensible order, respecting trend• Appropriate instructions and codings• Mindful questionnaire design
  50. 50. Gary LangerLanger Research Associatesglanger@langerresearch.comNYAAPOR 6/6/13