Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin


Published on

  • Be the first to comment

  • Be the first to like this

A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin

  1. 1. A Journalist’s Guide to SurveyResearch and Election PollsCliff Zukin *Rutgers UniversityAAPOR The American Association for Public Opinion ResearchWashington Press Club Briefing 9/24/12*This presentation reflects the author’s views. It has not been vetted or endorsed by AAPOR.
  2. 2. First, can we trust polls: 2004?
  3. 3. Can we trust polls: 2008?
  4. 4. What AAPOR Thinks You Should Know1. Who did the poll? Who paid for it?2. When was it done?3. Who was sampled?4. How were respondents contacted?5. What is sampling error?6. Why are data weighted?7. How is the question worded?8. How are the questions ordered?4
  5. 5. What You Want to Know1. How are margins of error determined, and whatdoes the error rate mean?2. To what extent are traditional polls beingreplaced by targeted cell phone or Internet polls?3. Discuss Internet polls and their validity.4. Any new polling techniques for 2012?5. Explain weighting, and how subjective it is, andhow to evaluate it.6. What are the most common errors made bypolitical journalists in writing about polls? 5
  6. 6. Our Roadmap(Through the Methods Box. See the handout. )• Sampling—why polls work: Good and Bad Samples• Developments and Challenges facing the SRprofession: cell phones, IVR (robo-polls), etc.• Question wording & ordering• Things you want to be wary of…SR for journalists• Election Polling—How it’s different & why polls differ6
  7. 7. SAMPLING: The Science of Polling
  8. 8. All Scientific Polling is Based on theNotion of Sampling• Bloody Whiners I’VE NEVER BEEN CALLED• Polls are estimates• A sample is drawn to represent an underlyingpopulation• The sample must be representative• If you don’t do this well, the rest doesn’t matter8
  9. 9. Telling Good Polls and Bad Polls ApartStarts with the Sample• Probability Samples are GOOD– Scientific—known chance of inclusion– Random, or thereabouts, selection– Generalizable from sample to population• Non-Probability Samples are BAD– Not representative: 1,200 people are not a sample of anything– Person in the street– ALL Internet opt-in surveys– Any self-selection9
  10. 10. Journalistic Standards• In order to represent the population statistically, a surveyshould be based on a probability sample.-NY Times• Methodologically, in all or nearly all cases we require aprobability sample, with high levels of coverage of acredible sampling frame. Self-selected or so-called“convenience” samples, including internet, e-mail, “blastfax,” call-in, street intercept, and non-probability mail-insamples do not meet our standards for validity andreliability, and we recommend against reporting them-ABC/Washington Post10
  11. 11. Probabilities, Better Known as The Odds:Why Polls are Accurate…We’re the HouseDiceTotalHow you can make this totalHow many ways todo itFor every 100 times, youshould get this outcomethis many X2 1,1 1 33 1,2; 2,1 2 54 1,3; 2,2; 3,1 3 85 1,4; 2,3; 3,2; 4,1 4 116 1,5; 2,4; 3,3; 4,2; 5,1 5 147 1,6; 2,5; 3,4; 4,3; 5,2; 6,1 6 178 2,6; 3,5; 4,4; 5,3; 6,2 5 149 3,6; 4,5; 5,4; 6,3 4 1110 4,6; 5,5; 6,4 3 811 5,6; 6,5 2 512 6,6 1 336 Total Outcomes 11
  12. 12. Dice Outcome(Or, the Central Limits Theorem)12
  13. 13. Probabilities: Why Polls are Accurateor “We’re the House”• Chances of getting a 7 = 17% average or mean(of 100 rolls (100%)/ 6 ways = 17%)• Chances of getting between 6 and 8 = 45%(14% + 17% +14% = 45%)• Chances of getting a 5 through 9 = 67%(11% + 14% + 17% +14% + 11% = 67%)• So, the chances of being within 4 points on eitherside of the expected (mean) is 95%. Or the marginof error is + 4 % points13
  14. 14. Probability(good) Samples Depend on Mathematical Axioms:The Central Limits Theorem & the Law of Large Numbers1. Life (or a sample of life) is distributed normally andgenerally fits a bell-shaped curve Central Limits2. The more observations you have, the more yourdistribution will approximate a bell-shaped curveLarge numbers3. After a certain point, sample size matters, butpopulation size doesn’t. Law of M&Ms14
  15. 15. At a Secret Location in Hacketstown NJ…15
  16. 16. …I Counted the M&Ms in the Last SlideColor NumberTan 4Brown 8Red 9Orange 10Green 7Yellow 2Total 4016
  17. 17. Relationship between Sample Sizeand Sampling Error17
  18. 18. • Sampling Error is a theoretical minimum• It is only one kind of error, but it is quantifiable• It applies not to the GAP between candidates, butto each point estimateObama 47 Romney 44Obama 47 ± 3 Romney 44 ± 3Obama 44 + 50 Romney 41 – 47MOE ± 3Sampling Error18
  19. 19. SO WHAT? Why is this Important?• INTERNET POLLS ARE NON-PROBABILITY SAMPLES(WITH ONE EXCEPTION)• It is not possible to calculate sampling erroron a Non-Probability Sample• And even with lipstick, a pig is still a pigIpsos Poll for Reuters DAILY ELECTION TRACKING 9/11/12:These are findings from an Ipsos poll conducted for Thomson Reuters from Sept 7-11, 2012. For the survey, a sample of 1,269 American registered voters (age 18and over) was interviewed online. The precision of the Reuters/Ipsos online polls ismeasured using a credibility interval. In this case, the poll has a credibility intervalof plus or minus 3.1 percentage points for all respondents.19
  20. 20. It’s Not Just Sampling Error:Total Survey Error Comprises 4 Sources• Sampling Error: Sample, not population surveyed• Coverage Error: Sample may not map to pop• Measurement Error: Question wording/ordering• Non-response Error: Many decline to be surveyed20
  21. 21. Trends in Survey Research
  22. 22. Trends in the Survey Research Industry• Cell phones• IVRs (Robo-Polls)• Fewer high-quality polls• Challenge of interviewing a representative sample• Increased cost of finding respondents and dialing22
  23. 23. *Data Source: CDC/NCHS National Health Interview SurveyPercentage of Wireless Only Adults 2001-200523
  24. 24. Percentage of Wireless Only Adults 2001-2011*Data Source: CDC/NCHS National Health Interview Survey 24
  25. 25. Percent of Adults Ages 18-29Source: Current Population Survey and Pew Research Center surveys25
  26. 26. RDD Samples without Cell Phones – Age Bias26
  27. 27. Phone Status of Adults, Dec 2011Among those households with telephones…•There are four times more “cell phone only” (32%) inthe population as “landline only” (8%)•Taken together, fully half of the public is “cell phoneonly” (32%) or “cell phone mostly” (18%)•Did you know: Most IVR Polls do not include cellphones in their samples? This is definitely a questionyou should ask before reporting27
  28. 28. Journalistic Standards: NY Times• Interactive Voice Response Polls — Interactive voice response(IVR) polls (also known as "robo-polls") employ an automated,recorded voice to call respondents who are asked to answerquestions by punching telephone keys.The Times does not publish IVR polls.• Internet Polls — Non-probability samples are commonly usedin Internet polls, call-in polls, blast e-mail polls and a variety ofothers. The entire population does not have an equal chanceof being contacted. Most Internet polls are based on panelsof self-selected respondents.The Times does not publish most Internet polls.28
  29. 29. Manipulating Public OpinionOr can you make a poll find anything you want?NO!BUT question wording and questionorder are extremely important29
  30. 30. Good Questions:1. Are simple, direct, clear to all, and avoid jargon2. Don’t presume information3. Are balanced4. Ask about only one thing5. Don’t tax respondent’s memory or cognitive ability30
  31. 31. Bad Questions:1. Are complex or presume information2. Are leading or unbalanced3. Are double-barreled or double negative4. Are loaded through emotional or red flag words5. Give biasing or unequal information in the Q stem31
  32. 32. Bad Questions:1. Are complex or presume information:Do you favor or oppose a 1 percent tax by the United Nations oninternational air travel which by itself would produce half of all therevenue the organization needs and a savings for all members,which for the United States would be about half a billion dollars ayear. The tax itself would add about a billion dollars to America’sinternational travel and air freight bills. Do you favor or oppose that?32% Strongly favor30% Somewhat favor11% Somewhat oppose25% Strongly oppose2% Dont know/No answer32
  33. 33. Bad Questions:1. Are complex or presume information:President Obama has announced changes to federal studentloan programs that would: allow some college graduates tolimit federal student loan repayments to 10% of theirdiscretionary income starting in January, two years before thecap was due to take effect under federal law; forgiveremaining debt on the federal loans after 20 years, five yearsearlier than under current law; and allow those with more thanone student loan to consolidate their debt. Do you approveor disapprove of these changes?33
  34. 34. Getting Real:Citizen Knowledge and Attentiveness• 17% follow “news about political figures and eventsin Washington very closely”• 28% named John Roberts as Chief Justice out offour choices (7/10)• 43% know Republicans control the House (11/11)• 53% Know it is the Republicans who want to reducethe size of governmentSource: Pew Research Center 34
  35. 35. Bad Questions:2. Are leading (or unbalanced):Are you in favor of the mandatory drug testing ofprofessional athletes? BIASEDRather,Do you favor or oppose the mandatory…UNBIASEDor… professional athletes, or not? UNBIASED35
  36. 36. Bad Questions:Totals may not add up to 100% due to rounding.2. Are leading (or unbalanced):Should the federal government increase its role in providingloans with a goal of making sure everyone who puts in theeffort to graduate from college can afford to do so?• It’s okay for you to ask leading questions of experts,but not for us to ask them of random folks36
  37. 37. Bad Questions:3. Are double-barreled or double negative:Did you vote in the 2008 and 2010 elections – yes or no?37
  38. 38. Bad Questions:3. Are double-barreled or double negative:Do you want to see less money spent on defense and moreon social problems?38
  39. 39. Bad Questions:3. Are double-barreled or double negative:(Agree or disagree) Should the federal government no longerbe involved in college loans, and instead leave that entirelyto the private sector?39
  40. 40. 4. Are loaded through emotional or red flag wordsFifty state legislatures have passed resolutions calling onCongress to pass the flag amendment and send it to thestates for ratification. Do you think members of Congress whomay personally oppose the amendment should vote in favorof it anyway so that “we the people” can decide the issue?Bad Questions:40
  41. 41. 4. Are loaded through emotional or red flag wordsDo you believe in killing unborn babies?Bad Questions:41
  42. 42. 5. Through giving one-sided informationIf it would result in increased opportunities for educating NewJersey citizens, would you favor or oppose building a new TVtransmitter at liberty science center?Bad Questions:42
  43. 43. 5. Through giving one-sided informationMany gambling experts believe that internet gambling willcontinue no matter what the government does to try to stop it.Do you agree or disagree that the federal government shouldallocate government resources and spend taxpayer moneytrying to stop adult Americans from gambling online?Balance is necessary:Some people say…while others say…Bad Questions:43
  44. 44. ALASKA, or (I’ll Ask her)• Zogby: for environmental interestsDo you think oil companies should be allowed to drill for oil inAmerica’s Arctic National Wildlife Refuge?Allow 38% Not allow 55%17 percentage points against drilling44
  45. 45. ALASKA, or (I’ll Ask her)45
  46. 46. ALASKA, or (I’ll Ask her)46
  47. 47. Maxims of Question Wording1. Question wording matters most when respondents know theleast2. Projections and hypotheticals are generally unreliable3. You can never really underestimate how much peopleknow about politics and government4. Giving information before asking the question is a double-edged sword5. It must be balanced--as easy to disagree as to agree6. No ONE question tells the story47
  48. 48. CONTEXT EFFECTSEach Question Affects the Following Ones• How important is Social Security to you?• Do you think Social Security will be there when youneed it?• Are you worried about private investment optionsfor Social Security?• What is the most important problem facing thecountry?48
  49. 49. WHAT’S THE TOPIC HERE….1. “The First Amendment of the United States applies toeveryone regardless of gender, race, religion, ageprofession, or point of view.” (96%)2. “The First Amendment protects the right of individuals tocreate a private organization consisting of a specificgroup of people based on age, gender, race, ethnicity,or interest.” (77%)3. “The First Amendment protects the right oforganizations like the Boy Scouts, the Girl Scouts, andthe National Association for the Advancement ofColored People to exist.” (91%)49
  50. 50. IT’S BEEN IN THE NEWS RECENTLY….4. “Individuals have a right to join a private group, club,or organization that consists of people who share thesame interests and personal backgrounds as they do ifthey so desire.” (91%)5. “Private organizations that are not funded by thegovernment should be allowed to decide whobecomes a member and who does not become amember on their own, without being forced to takeinput from other outside people or organizations.”(76%)50
  51. 51. The Masters Polling Co…6. “Private clubs that have members only of the samegender are simply a harmless way for similar individualsto get together and associate with each other.” (71%)23. “Although currently there are no women members ofthe Augusta National Golf Club, the Golf Club doesallow women to play on their golf course, and visit thecourse for the Masters Tournament. In other words,women are welcome to visit the Club and they oftenplay as guests.”“Knowing this, would say you support or oppose the AugustaNational Golf Clubs decision to keep their membership policyas it is?” (60%)51
  52. 52. Some Common Problems inReporting Polls
  53. 53. What Do I Write if it’s 50 – 46?AP Style Book:•If the difference between the candidates is more than twice thesampling error margin, then the poll says one candidate is leading.•If the difference is less than the sampling error margin, the poll saysthat the race is close, that the candidates are "about even."– (Do not use the term "statistical dead heat," which is inaccurate if there is anydifference between the candidates; if the poll finds the candidates are tied, saytheyre tied.)•If the difference is at least equal to the sampling error but no morethan twice the sampling error, then one candidate can be said tobe "apparently leading" or "slightly ahead" in the race.53
  54. 54. Be Wary of the “Slight” or “Modest” TrendsFrom the January 14, 2005, Hartford Courant:• …Rells stellar approval rating crept even higher ina poll released Thursday, rising from 80 percent to83 percent in the first survey since her cancersurgery and State of the State address….• Telephone poll of 1,287 voters from 1/7-10/05.Margin of error of +/- 2.7 percentage points.54
  55. 55. Was Jesus married?Beware Small Groups and Subgroups:Size Does Matter55
  56. 56. Inadequate Sample Size:Independent Soccer Mom Voters in New Jersey56
  57. 57. What’s the Difference?7 797 79 911121012151114 14170246810121416181987 1988 1990 1994 1997 1999 2002 2003Political and Policy Attitudes Social and Personal AttitudesRepublicans and Democrats are Further Apart than Ever:Average Difference in Republican and Democratic Attitudes57
  58. 58. An Alternative Perspective:01020304050607080901001987 1988 1990 1994 1997 1999 2002 2003Political and Policy Attitudes Social and Personal AttitudesRepublicans and Democrats are Further Apart than Ever?58
  59. 59. Comparing Two Different Polls“After a day of action at the Republican convention in Tampa,GOP presidential candidate Mitt Romney is looking better tovoters. Romney’s image has received a five-point bump afterthe convention’s first day, according to data presented at aWednesday breakfast sponsored by The Hill at the law offices ofHolland & Knight.”“Romney had a 43% favorable and 44% unfavorable rating innine battleground states heading into the convention,according to the average compiled by Real Clear Politics. Asurvey conducted by Wilson Perkins Allen Opinion Research innine battleground states Tuesday evening found Romney’sfavorable rating among likely voters to have jumped to 48%. Hisunfavorable rating dipped to 39%.”59
  60. 60. Beware the Costless Question:Would you favor or oppose preserving more openland in New Jersey?• Favor• Oppose• Depends how much (VOL)• Don´t Know / Refused (VOL)60
  61. 61. Don’t Report Answers to Silly Questions“Sixty-seven percent of American Catholicsbelieve John Paul II was the best pope thechurch has ever had, according to a newCNN poll of 254 Catholics.”61
  62. 62. About WEIGHTING• It’s necessary in almost all surveys• Everyone does it• It corrects for the problem of not interviewingpeople in the sample in correct proportion totheir size in the population62
  63. 63. Weighting ExampleEducation PopulationLess than High School 10High School 25Some College 35College Graduate 30Sample5103550Weight2.
  64. 64. Limitations of Weighting• You can only weight to KNOWN populationparameters• Weighting to “attitudes” (like Party ID) is risky—theyare not fixed attributes• You can’t weight to something in the future (liketurnout)64
  65. 65. Election PollingA Special Case
  66. 66. ELECTION POLLING: A Special Case• Time and field dates/procedures• Type of sample used: random or listed• Resp. selection/screening; Identifying likely voters• Question wording and ordering• Weighting66
  67. 67. Time, Field Dates and Procedures• Polls are a snapshot (yada yada)• Number of days in the field:– Callbacks and refusal conversions• Tracking polls/rolling averages67
  68. 68. • Random digit dialing (RDD)• Addressed based sampling (ABS)• Listed based sample (LBS)Types of Samples:Advantages and Disadvantages68
  69. 69. Source: Pew Research Center69
  70. 70. Respondent Selection, Screening andDetermining “Likely Voters”• Hard screen to get into the survey, or remove low likelihood ofvoting respondents after the fact?• The problem: There is a consistent over-report of intentions tovote by the publicWhat do you do if 80percent of your eligiblevoters tell you they willdefinitely vote for President,but you’re looking at thetable on the right?Election Year Percent ofEligible Voters1992 60.61996 52.62000 54.22004 60.12008 61.170
  71. 71. Pew Likely Voter Questions• How much thought have you given to the coming presidential election?• How closely have you been following news about the candidates?• Do you plan to vote in the presidential election? How certain are you that you willvote?• Rate your chance of voting in November on a scale of 10 to 1, with 10 being“definitely will vote” and 1 “definitely will not vote.”• How often do you follow what’s going on in government and public affairs?• Have you ever voted in your precinct or election district?• How often would you say you vote?• Do you happen to know where people in your neighborhood go to vote?• In the last election, did things come up that kept you from voting or did you vote?71
  72. 72. Likely Voter ComputationScore Percent (%)9 408 87 76 125 204 153 42 31 10 0Target72
  73. 73. Likely Voter Vote DivisionCandidate Turnout(All)Obama 50Romney 40Don’t know 10Margin 10Top 67% Top 55%50 5343 487 47 573
  74. 74. Question wording and ordering• Asking the vote question: The presidentialcandidates and ….?• …VP candidates,• …party labels,• …randomized names,• (3rdpty if applic)?• Where the vote question is put: What came beforeit that it will be reactive to?74
  75. 75. Weighting• To what parameters, even on demographics?• Weighting on party is controversial; Best practices”says this is NOT a good idea• But what’s a pollster to do if faced with:75
  76. 76. The Special Case of Party IDParty ID Phone IVR InternetDemocrat 33% 36% 46%Independent 35% 29% 15%Republican 30% 34% 36%Totals 98% 99% 97%* Mark Blumenthal @ Huffington Post August 24, 201276
  77. 77. Why Election Polls May Vary by a FewPercentage Points• Sampling error• Length of field period• Live interviewers vs. IVR• Type of sample used• Mode of administration• Respondent selection• Likely voter indices• Question wording &ordering• WeightingActually, it’sa wonderthey are asclose asthey are!77
  78. 78. Know Your “Rights”AAPOR Minimum Disclosure Standards• Name of the surveysponsor• Name of organizationthat conducted thesurvey• Exact wording ofquestions being released• Definition of populationunder study• Description of samplingframe used to representpopulation under study• Explanation ofrespondent selection• Total sample size• Method or mode of datacollection• Dates and location ofdata collection• Estimates of samplingerror (if appropriate)• Description of dataweighting• Use of sub-groupsdisclosed78
  79. 79. We’re (not from the government) Here to HELP• AAPOR– 3 Presidents• Paul Lavrakas President• Scott Keeter Past President• Rob Santos President-elect– Rapid Response (Election Polling) Team• Quin Monson• Mike Traugott• Rob Daves• Cliff Zukin zukin@ejb.rutgers.edu79