A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin

A Journalist’s Guide to Survey
Research and Election Polls
Cliff Zukin *
Rutgers University
AAPOR The American Association for Public Opinion Research
Washington Press Club Briefing 9/24/12
*This presentation reflects the author’s views. It has not been vetted or endorsed by AAPOR.

First, can we trust polls: 2004?
http://www.realclearpolitics.com/epolls/2004/president/us/general_election_bush_vs_kerry-939.html
2

Can we trust polls: 2008?
http://www.realclearpolitics.com/epolls/2008/president/national.html
3

What AAPOR Thinks You Should Know
1. Who did the poll? Who paid for it?
2. When was it done?
3. Who was sampled?
4. How were respondents contacted?
5. What is sampling error?
6. Why are data weighted?
7. How is the question worded?
8. How are the questions ordered?
4

What You Want to Know
1. How are margins of error determined, and what
does the error rate mean?
2. To what extent are traditional polls being
replaced by targeted cell phone or Internet polls?
3. Discuss Internet polls and their validity.
4. Any new polling techniques for 2012?
5. Explain weighting, and how subjective it is, and
how to evaluate it.
6. What are the most common errors made by
political journalists in writing about polls? 5

Our Roadmap
(Through the Methods Box. See the handout. )
• Sampling—why polls work: Good and Bad Samples
• Developments and Challenges facing the SR
profession: cell phones, IVR (robo-polls), etc.
• Question wording & ordering
• Things you want to be wary of…SR for journalists
• Election Polling—How it’s different & why polls differ
6

SAMPLING: The Science of Polling
http://faculty.elgin.edu/dkernler/statistics/ch01/3-1.html
7

All Scientific Polling is Based on the
Notion of Sampling
• Bloody Whiners I’VE NEVER BEEN CALLED
• Polls are estimates
• A sample is drawn to represent an underlying
population
• The sample must be representative
• If you don’t do this well, the rest doesn’t matter
8

Telling Good Polls and Bad Polls Apart
Starts with the Sample
• Probability Samples are GOOD
– Scientific—known chance of inclusion
– Random, or thereabouts, selection
– Generalizable from sample to population
• Non-Probability Samples are BAD
– Not representative: 1,200 people are not a sample of anything
– Person in the street
– ALL Internet opt-in surveys
– Any self-selection
9

Journalistic Standards
• In order to represent the population statistically, a survey
should be based on a probability sample.
-NY Times
• Methodologically, in all or nearly all cases we require a
probability sample, with high levels of coverage of a
credible sampling frame. Self-selected or so-called
“convenience” samples, including internet, e-mail, “blast
fax,” call-in, street intercept, and non-probability mail-in
samples do not meet our standards for validity and
reliability, and we recommend against reporting them
-ABC/Washington Post
10

Probabilities, Better Known as The Odds:
Why Polls are Accurate…We’re the House
Dice
Total
How you can make this total
How many ways to
do it
For every 100 times, you
should get this outcome
this many X
2 1,1 1 3
3 1,2; 2,1 2 5
4 1,3; 2,2; 3,1 3 8
5 1,4; 2,3; 3,2; 4,1 4 11
6 1,5; 2,4; 3,3; 4,2; 5,1 5 14
7 1,6; 2,5; 3,4; 4,3; 5,2; 6,1 6 17
8 2,6; 3,5; 4,4; 5,3; 6,2 5 14
9 3,6; 4,5; 5,4; 6,3 4 11
10 4,6; 5,5; 6,4 3 8
11 5,6; 6,5 2 5
12 6,6 1 3
36 Total Outcomes 11

Dice Outcome
(Or, the Central Limits Theorem)
12

Probabilities: Why Polls are Accurate
or “We’re the House”
• Chances of getting a 7 = 17% average or mean
(of 100 rolls (100%)/ 6 ways = 17%)
• Chances of getting between 6 and 8 = 45%
(14% + 17% +14% = 45%)
• Chances of getting a 5 through 9 = 67%
(11% + 14% + 17% +14% + 11% = 67%)
• So, the chances of being within 4 points on either
side of the expected (mean) is 95%. Or the margin
of error is + 4 % points
13

Probability(good) Samples Depend on Mathematical Axioms:
The Central Limits Theorem & the Law of Large Numbers
1. Life (or a sample of life) is distributed normally and
generally fits a bell-shaped curve Central Limits
2. The more observations you have, the more your
distribution will approximate a bell-shaped curve
Large numbers
3. After a certain point, sample size matters, but
population size doesn’t. Law of M&Ms
14

At a Secret Location in Hacketstown NJ…
15

…I Counted the M&Ms in the Last Slide
Color Number
Tan 4
Brown 8
Red 9
Orange 10
Green 7
Yellow 2
Total 40
16

Relationship between Sample Size
and Sampling Error
17

• Sampling Error is a theoretical minimum
• It is only one kind of error, but it is quantifiable
• It applies not to the GAP between candidates, but
to each point estimate
Obama 47 Romney 44
Obama 47 ± 3 Romney 44 ± 3
Obama 44 + 50 Romney 41 – 47
MOE ± 3
Sampling Error
18

SO WHAT? Why is this Important?
• INTERNET POLLS ARE NON-PROBABILITY SAMPLES
(WITH ONE EXCEPTION)
• It is not possible to calculate sampling error
on a Non-Probability Sample
• And even with lipstick, a pig is still a pig
Ipsos Poll for Reuters DAILY ELECTION TRACKING 9/11/12:
These are findings from an Ipsos poll conducted for Thomson Reuters from Sept 7-
11, 2012. For the survey, a sample of 1,269 American registered voters (age 18
and over) was interviewed online. The precision of the Reuters/Ipsos online polls is
measured using a credibility interval. In this case, the poll has a credibility interval
of plus or minus 3.1 percentage points for all respondents.
19

It’s Not Just Sampling Error:
Total Survey Error Comprises 4 Sources
• Sampling Error: Sample, not population surveyed
• Coverage Error: Sample may not map to pop
• Measurement Error: Question wording/ordering
• Non-response Error: Many decline to be surveyed
20

Trends in the Survey Research Industry
• Cell phones
• IVRs (Robo-Polls)
• Fewer high-quality polls
• Challenge of interviewing a representative sample
• Increased cost of finding respondents and dialing
22

*Data Source: CDC/NCHS National Health Interview Survey
Percentage of Wireless Only Adults 2001-2005
23

Percentage of Wireless Only Adults 2001-2011
*Data Source: CDC/NCHS National Health Interview Survey 24

Percent of Adults Ages 18-29
Source: Current Population Survey and Pew Research Center surveys
25

RDD Samples without Cell Phones – Age Bias
26

Phone Status of Adults, Dec 2011
Among those households with telephones…
•There are four times more “cell phone only” (32%) in
the population as “landline only” (8%)
•Taken together, fully half of the public is “cell phone
only” (32%) or “cell phone mostly” (18%)
•Did you know: Most IVR Polls do not include cell
phones in their samples? This is definitely a question
you should ask before reporting
27

Journalistic Standards: NY Times
• Interactive Voice Response Polls — Interactive voice response
(IVR) polls (also known as "robo-polls") employ an automated,
recorded voice to call respondents who are asked to answer
questions by punching telephone keys.
The Times does not publish IVR polls.
• Internet Polls — Non-probability samples are commonly used
in Internet polls, call-in polls, blast e-mail polls and a variety of
others. The entire population does not have an equal chance
of being contacted. Most Internet polls are based on panels
of self-selected respondents.
The Times does not publish most Internet polls.
28

Manipulating Public Opinion
Or can you make a poll find anything you want?
NO!
BUT question wording and question
order are extremely important
29

Good Questions:
1. Are simple, direct, clear to all, and avoid jargon
2. Don’t presume information
3. Are balanced
4. Ask about only one thing
5. Don’t tax respondent’s memory or cognitive ability
30

Bad Questions:
1. Are complex or presume information
2. Are leading or unbalanced
3. Are double-barreled or double negative
4. Are loaded through emotional or red flag words
5. Give biasing or unequal information in the Q stem
31

Bad Questions:
1. Are complex or presume information:
Do you favor or oppose a 1 percent tax by the United Nations on
international air travel which by itself would produce half of all the
revenue the organization needs and a savings for all members,
which for the United States would be about half a billion dollars a
year. The tax itself would add about a billion dollars to America’s
international travel and air freight bills. Do you favor or oppose that?
32% Strongly favor
30% Somewhat favor
11% Somewhat oppose
25% Strongly oppose
2% Don't know/No answer
32

Bad Questions:
1. Are complex or presume information:
President Obama has announced changes to federal student
loan programs that would: allow some college graduates to
limit federal student loan repayments to 10% of their
discretionary income starting in January, two years before the
cap was due to take effect under federal law; forgive
remaining debt on the federal loans after 20 years, five years
earlier than under current law; and allow those with more than
one student loan to consolidate their debt. Do you approve
or disapprove of these changes?
33

Getting Real:
Citizen Knowledge and Attentiveness
• 17% follow “news about political figures and events
in Washington very closely”
• 28% named John Roberts as Chief Justice out of
four choices (7/10)
• 43% know Republicans control the House (11/11)
• 53% Know it is the Republicans who want to reduce
the size of government
Source: Pew Research Center 34

Bad Questions:
2. Are leading (or unbalanced):
Are you in favor of the mandatory drug testing of
professional athletes? BIASED
Rather,
Do you favor or oppose the mandatory…UNBIASED
or
… professional athletes, or not? UNBIASED
35

Bad Questions:
Totals may not add up to 100% due to rounding.
2. Are leading (or unbalanced):
Should the federal government increase its role in providing
loans with a goal of making sure everyone who puts in the
effort to graduate from college can afford to do so?
• It’s okay for you to ask leading questions of experts,
but not for us to ask them of random folks
36

Bad Questions:
3. Are double-barreled or double negative:
Did you vote in the 2008 and 2010 elections – yes or no?
37

Bad Questions:
Do you want to see less money spent on defense and more
on social problems?
38

Bad Questions:
(Agree or disagree) Should the federal government no longer
be involved in college loans, and instead leave that entirely
to the private sector?
39

Fifty state legislatures have passed resolutions calling on
Congress to pass the flag amendment and send it to the
states for ratification. Do you think members of Congress who
may personally oppose the amendment should vote in favor
of it anyway so that “we the people” can decide the issue?
Bad Questions:
40

Do you believe in killing unborn babies?
Bad Questions:
41

5. Through giving one-sided information
If it would result in increased opportunities for educating New
Jersey citizens, would you favor or oppose building a new TV
transmitter at liberty science center?
Bad Questions:
42

5. Through giving one-sided information
Many gambling experts believe that internet gambling will
continue no matter what the government does to try to stop it.
Do you agree or disagree that the federal government should
allocate government resources and spend taxpayer money
trying to stop adult Americans from gambling online?
Balance is necessary:
Some people say…while others say…
Bad Questions:
43

ALASKA, or (I’ll Ask her)
• Zogby: for environmental interests
Do you think oil companies should be allowed to drill for oil in
America’s Arctic National Wildlife Refuge?
Allow 38% Not allow 55%
17 percentage points against drilling
44

45

46

Maxims of Question Wording
1. Question wording matters most when respondents know the
least
2. Projections and hypotheticals are generally unreliable
3. You can never really underestimate how much people
know about politics and government
4. Giving information before asking the question is a double-
edged sword
5. It must be balanced--as easy to disagree as to agree
6. No ONE question tells the story
47

CONTEXT EFFECTS
Each Question Affects the Following Ones
• How important is Social Security to you?
• Do you think Social Security will be there when you
need it?
• Are you worried about private investment options
for Social Security?
• What is the most important problem facing the
country?
48

WHAT’S THE TOPIC HERE….
1. “The First Amendment of the United States applies to
everyone regardless of gender, race, religion, age
profession, or point of view.” (96%)
2. “The First Amendment protects the right of individuals to
create a private organization consisting of a specific
group of people based on age, gender, race, ethnicity,
or interest.” (77%)
3. “The First Amendment protects the right of
organizations like the Boy Scouts, the Girl Scouts, and
the National Association for the Advancement of
Colored People to exist.” (91%)
49

IT’S BEEN IN THE NEWS RECENTLY….
4. “Individuals have a right to join a private group, club,
or organization that consists of people who share the
same interests and personal backgrounds as they do if
they so desire.” (91%)
5. “Private organizations that are not funded by the
government should be allowed to decide who
becomes a member and who does not become a
member on their own, without being forced to take
input from other outside people or organizations.”
(76%)
50

The Masters Polling Co…
6. “Private clubs that have members only of the same
gender are simply a harmless way for similar individuals
to get together and associate with each other.” (71%)
23. “Although currently there are no women members of
the Augusta National Golf Club, the Golf Club does
allow women to play on their golf course, and visit the
course for the Masters Tournament. In other words,
women are welcome to visit the Club and they often
play as guests.”
“Knowing this, would say you support or oppose the Augusta
National Golf Clubs decision to keep their membership policy
as it is?” (60%)
51

Some Common Problems in
Reporting Polls

What Do I Write if it’s 50 – 46?
AP Style Book:
•If the difference between the candidates is more than twice the
sampling error margin, then the poll says one candidate is leading.
•If the difference is less than the sampling error margin, the poll says
that the race is close, that the candidates are "about even."
– (Do not use the term "statistical dead heat," which is inaccurate if there is any
difference between the candidates; if the poll finds the candidates are tied, say
they're tied.)
•If the difference is at least equal to the sampling error but no more
than twice the sampling error, then one candidate can be said to
be "apparently leading" or "slightly ahead" in the race.
53

Be Wary of the “Slight” or “Modest” Trends
From the January 14, 2005, Hartford Courant:
• …Rell's stellar approval rating crept even higher in
a poll released Thursday, rising from 80 percent to
83 percent in the first survey since her cancer
surgery and State of the State address….
• Telephone poll of 1,287 voters from 1/7-10/05.
Margin of error of +/- 2.7 percentage points.
54

Was Jesus married?
Beware Small Groups and Subgroups:
Size Does Matter
55

Inadequate Sample Size:
Independent Soccer Mom Voters in New Jersey
56

What’s the Difference?
7 7
9
7 7
9 9
11
12
10
12
15
11
14 14
17
0
2
4
6
8
10
12
14
16
18
1987 1988 1990 1994 1997 1999 2002 2003
Political and Policy Attitudes Social and Personal Attitudes
Republicans and Democrats are Further Apart than Ever:
Average Difference in Republican and Democratic Attitudes
57

An Alternative Perspective:
0
10
20
30
40
50
60
70
80
90
100
1987 1988 1990 1994 1997 1999 2002 2003
Political and Policy Attitudes Social and Personal Attitudes
Republicans and Democrats are Further Apart than Ever?
58

Comparing Two Different Polls
“After a day of action at the Republican convention in Tampa,
GOP presidential candidate Mitt Romney is looking better to
voters. Romney’s image has received a five-point bump after
the convention’s first day, according to data presented at a
Wednesday breakfast sponsored by The Hill at the law offices of
Holland & Knight.”
“Romney had a 43% favorable and 44% unfavorable rating in
nine battleground states heading into the convention,
according to the average compiled by Real Clear Politics. A
survey conducted by Wilson Perkins Allen Opinion Research in
nine battleground states Tuesday evening found Romney’s
favorable rating among likely voters to have jumped to 48%. His
unfavorable rating dipped to 39%.”
59

Beware the Costless Question:
Would you favor or oppose preserving more open
land in New Jersey?
• Favor
• Oppose
• Depends how much (VOL)
• Don´t Know / Refused (VOL)
60

Don’t Report Answers to Silly Questions
“Sixty-seven percent of American Catholics
believe John Paul II was the best pope the
church has ever had, according to a new
CNN poll of 254 Catholics.”
61

About WEIGHTING
• It’s necessary in almost all surveys
• Everyone does it
• It corrects for the problem of not interviewing
people in the sample in correct proportion to
their size in the population
62

Weighting Example
Education Population
Less than High School 10
High School 25
Some College 35
College Graduate 30
Sample
5
10
35
50
Weight
2.0
2.5
1.0
0.6
63

Limitations of Weighting
• You can only weight to KNOWN population
parameters
• Weighting to “attitudes” (like Party ID) is risky—they
are not fixed attributes
• You can’t weight to something in the future (like
turnout)
64

Election Polling
A Special Case

ELECTION POLLING: A Special Case
• Time and field dates/procedures
• Type of sample used: random or listed
• Resp. selection/screening; Identifying likely voters
• Question wording and ordering
• Weighting
66

Time, Field Dates and Procedures
• Polls are a snapshot (yada yada)
• Number of days in the field:
– Callbacks and refusal conversions
• Tracking polls/rolling averages
67

• Random digit dialing (RDD)
• Addressed based sampling (ABS)
• Listed based sample (LBS)
Types of Samples:
Advantages and Disadvantages
68

Source: Pew Research Center
69

Respondent Selection, Screening and
Determining “Likely Voters”
• Hard screen to get into the survey, or remove low likelihood of
voting respondents after the fact?
• The problem: There is a consistent over-report of intentions to
vote by the public
What do you do if 80
percent of your eligible
voters tell you they will
definitely vote for President,
but you’re looking at the
table on the right?
Election Year Percent of
Eligible Voters
1992 60.6
1996 52.6
2000 54.2
2004 60.1
2008 61.1
70

Pew Likely Voter Questions
• How much thought have you given to the coming presidential election?
• How closely have you been following news about the candidates?
• Do you plan to vote in the presidential election? How certain are you that you will
vote?
• Rate your chance of voting in November on a scale of 10 to 1, with 10 being
“definitely will vote” and 1 “definitely will not vote.”
• How often do you follow what’s going on in government and public affairs?
• Have you ever voted in your precinct or election district?
• How often would you say you vote?
• Do you happen to know where people in your neighborhood go to vote?
• In the last election, did things come up that kept you from voting or did you vote?
71

Likely Voter Computation
Score Percent (%)
9 40
8 8
7 7
6 12
5 20
4 15
3 4
2 3
1 1
0 0
Target
72

Likely Voter Vote Division
Candidate Turnout
(All)
Obama 50
Romney 40
Don’t know 10
Margin 10
Top 67% Top 55%
50 53
43 48
7 4
7 5
73

Question wording and ordering
• Asking the vote question: The presidential
candidates and ….?
• …VP candidates,
• …party labels,
• …randomized names,
• (3rd
pty if applic)?
• Where the vote question is put: What came before
it that it will be reactive to?
74

Weighting
• To what parameters, even on demographics?
• Weighting on party is controversial; Best practices”
says this is NOT a good idea
• But what’s a pollster to do if faced with:
75

The Special Case of Party ID
Party ID Phone IVR Internet
Democrat 33% 36% 46%
Independent 35% 29% 15%
Republican 30% 34% 36%
Totals 98% 99% 97%
* Mark Blumenthal @ Huffington Post August 24, 2012
76

Why Election Polls May Vary by a Few
Percentage Points
• Sampling error
• Length of field period
• Live interviewers vs. IVR
• Type of sample used
• Mode of administration
• Respondent selection
• Likely voter indices
• Question wording &
ordering
• Weighting
Actually, it’s
a wonder
they are as
close as
they are!
77

Know Your “Rights”
AAPOR Minimum Disclosure Standards
• Name of the survey
sponsor
• Name of organization
that conducted the
survey
• Exact wording of
questions being released
• Definition of population
under study
• Description of sampling
frame used to represent
population under study
• Explanation of
respondent selection
• Total sample size
• Method or mode of data
collection
• Dates and location of
data collection
• Estimates of sampling
error (if appropriate)
• Description of data
weighting
• Use of sub-groups
disclosed
78

We’re (not from the government) Here to HELP
• AAPOR
– 3 Presidents
• Paul Lavrakas pjl@hughes.net President
• Scott Keeter skeeter@pewresearch.org Past President
• Rob Santos rsantos@urban.org President-elect
– Rapid Response (Election Polling) Team
• Quin Monson quin.monson@byu.edu
• Mike Traugott mtrau@umich.edu
• Rob Daves rob@davesandassociates.com
• Cliff Zukin zukin@ejb.rutgers.edu
79

A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (20)

Similar to A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin

Similar to A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin (20)

A Journalist’s Guide to Survey Research and Election Polls by Cliff Zuskin