Reporting only sampling error gives a misleading impression of scientific precision of survey results. Because it is the only easily quantifiable source of survey error, journalists like to report it, but saying the margin of error in the survey is +/- 3% gives a misleading impression of scientific precision about surveys because it ignores all other sources of error which are not quantifiable and usually go unreported.
Could also point out that statistical sampling theory was based, in part, on a Guinness brewmaster’s (Oxford graduate, William Gosset, who had a degree in chemistry and mathematics) method for “sampling” yeast cultures from a jar to estimate whether they would produce excellent beer worthy of the Guinness name! The application to public opinion is fairly straightforward. We want to find out about the opinions of the public at large, but we can only afford to study a small sample of people. We’re then faced with Gosset's Guinness problem: How well does this sample estimate what is going on in the larger population?
475 media effects methods 2012 up
Mark PeffleyPS 475, Politics & Media
What types of influence (effects) by what types of media on what types of behavior (attitudes & behaviors)?
Political attitudes toward a variety of different objects ▪ Political objects: Candidates, Issues, Government, Political trust, Parties, etc. ▪ Non-Political objects: Social Groups (e.g., women, Latinos, seniors). ▪ Cognition (e.g., stereotypes) & Affect (e.g., prejudice, emotions) Political behavior ▪ Voting decisions, turnout, civic engagement, etc.
News versus Entertainment (e.g., Biden on “Will & Grace”) Free (news) versus Paid (ads) News Medium: broadcast, newsprint, radio, Internet, social media Paid Medium: door-to-door, leaflets, direct mail, electronic mail, phone banks, ads Note: Much wider diversity of media & outlets is a huge challenge for media scholars
Reinforcement of existing attitudes Attitude change (persuasion) Political learning Subtle media effects Framing Priming Agenda setting
Trade-off in internal & external validity (next slide) Survey methods (dominant in Political Science) Experiments (dominant in Psychology) Triangulation: Methodological pluralism to enhance external & internal validity of findings Both: Survey & Experiments Lab Experiments (using adult subjects) Field Experiments (may talk about later) Survey experiments (a hybrid) Internet surveys (experiments via survey)
Trade-off between two types of validity: external versus internal validity : Can we state that the independent variables caused the dependent variable or are there confounding factors that make such conclusions tenuous? : Can we generalize the results of the current study across people, settings, and outcomes? : External validity... tend[s] to preoccupy critics of experiments. This near obsession... tend[s] to be used to dismiss experiments....
Surveys Lab Experiments Depth Interviews Focus Groups Strengths External validity: Internal validity: Insights: Insights: Generalize findings Identifying cause-and- •“Soak & poke” in long Beyond superficial from representative effect relationships via interviews with a handful responses sample to random assign. of of respondents to see How people think population participants to diff. how they think and why How different “Natural” setting treatments or no (e.g., about Will & Grace). groups think treatment •Allow subjects to Social influences Less expensive “reveal” meaning vs. Less expensive: inferring it from simple quick & dirty survey responses. •Less expensiveWeaknesses Internal validity: External validity: External & Internal External & Internal validity validity Identifying cause- Unrepresentativeness &-effect of subjects? Generalizability of relationships Laboratory setting? “sample,” social Superficial Unrealistic dynamics, setting. responses procedures, stimuli? Response bias (social desirability) Expensive
1. Is it random? Is the sample some variant of a probability (random) sample? Does everyone in the population have an equal chance of being selected? a) If not, the sample is not representative of the population; it is biased (e.g., Kinsey Institute, Howard Stern & SLOP polls, Time Magazine’s “Person of the Century” poll). Bad surveys. b) The results of a non-random sample cannot be generalized to the population.2. Is it large enough? Is the random sample large enough to avoid large amounts of sampling error? (is it > 600?)3. Is the polling firm reputable? Most reputable polls rely on some variant of a random sample. Are there “house” effects from surveys that lean Republican or Democratic? (see Nate Silver)4. Sampling errors are just the "tip of the iceberg" in terms of problems or errors with public opinion polls, and tend to give the reader a false sense of the accuracy of polling results.
NYT: “The nationwide telephone poll was conducted Sept. 10 to Sept. 14 with 881 registered voters and has a margin of sampling error of plus or minus 3 percentage points.”
In 1908, William Sealy Gosset (1876-1937), invented a statistical test that would change the sciences.Lesson: Good beer & good science gotogether!
Man of the Century: Adolf Hitler and Eric Cartman (Time) To usher in the New Millennium and say goodbye to the old, Time magazine sponsored an online poll to name the Person of the Century. At one point late in the balloting, Jesus Christ led the list, followed by Adolf Hitler and Eric Cartman (a character on the "South Park" cartoon show who finished 14th in Peoples Beautiful People Poll). Everybody Loves Tom! (Parade) 2005 was an Annus Horribilis for actor Tom Cruise. There was that unfortunate bit of couch-bouncing on the Oprah Winfrey show and his publicized bashing of Brook Shields for using physician-prescribed drugs to treat her depression. Parade.com decided to conduct an online poll to see if people thought Tom had only himself to blame for his troubles or whether it was the media’s fault. A whopping 84% of respondents blamed the media. “We at Parade found this a little bit fishy—so we did some investigating. We found out more than 14,000 (of the 18,000+ votes) that came in, were cast from only 10 computers! Furthermore, there was one computer responsible for nearly 8,400 votes alone, all blaming the media for Toms troubles. We also discovered that at least two other machines were the sources of inordinate numbers of votes….There is even a chance they wrote a special "bot" program for the sole purpose of skewing the results, rather than casting the votes by hand on a computer.” Girls Just Want to Have Too Much Fun (AMA) In March of 2006, the American Medical Association reported disturbing rates of binge drinking and unprotected sex among college women during spring break. The report was based on what the researchers claimed was a survey of “a random sample” of 644 women. The survey results were breathlessly reported on the Today Show, the CBS Early Show, and hundreds of reports followed on local television and radio newscasts. The findings also were reported in the Time magazine, and in a chart that ran in the New York Times. One problem: The sample was not random. The results were based on only women who volunteered to answer the question as part of an online survey panel. Only about a quarter of these women had ever gone on a spring break trip. The Times eventually published a correction explaining the misrepresentation.
Declining “coverage” in an age of increased cell phone use, caller id, and fewer land-lines. Move to Internet surveys. Causal Inference! Move to Internet surveys as online experiments?
Some correlations… More people die in cities with more doctors. ▪ What this means: Doctors are dangerous, like zombies! Students with more clothes have higher SAT scores. ▪ Buy more clothes to get into a good school! Those with long hair live longer than those with short hair. ▪ Never cut your hair!
Meese commission on pornography (1985) Upward trend in availability of pornographic material is associated with an increase in crime and violence generally in North America. ▪ Is there a directionality problem? ▪ Is there a possible third variable problem? ▪ Incidence of violent crime will positively correlate with anything that increased during the same period of time. ▪ Correlation between the incidence of rape and membership in the Southern Baptist church was very high (+.96) during the same time period. ▪ Correlation between Halloween and moral decay.
Survey research as correlational: are two or more factors naturally associated in the real world? Method of choice among traditional political scientists. Strength is external validity (generalizing findings of the sample to the population); Weakness is internal validity (detecting cause & effect), ▪ which is especially problematic for cross-sectional, one-shot surveys. Experimental research: manipulating some factor to see its effect on another factor. Method of choice among psychologists Strength is internal validity, the gold standard for understanding causation; Weakness is external validity, ▪ but too often this is taken as an excuse to reject experiments by “old school” political scientists.
Do attack (negative) ads discourage turnout? Survey research answer: No, recall of attack ads is actually associated with greater turnout, so voters are mobilized, not demobilized. ▪ Problem: Recall of attack ads is highly unreliable. And people who forget seeing an ad are less politically engaged, even though they can be influenced by such ads. Thus, the association between recall of ads and turnout is a spurious artifact.
Recall Recall of Ads Political of Ads Engagement Turnout Turnout Actual: the association is spurious—due to common Hypothesized dependence on Politicalby survey research Engagement.
Experimental answer: Subjects who viewed attack ads were 2.5% less likely to vote than the control group (who viewed no ads) & 5% less likely to turnout than those viewing positive ads. The ads were a single 30-second political ad embedded in a 15 minute newscast.
Subjects randomly assigned to 4 different versions of the ad varied by: • Candidate (Feinstein vs. Wilson) • Ad Tone (Positive vs. Negative) Feinstein, Feinstein, Pos. Neg. Wilson, Wilson, Pos. Neg.Product comparison versus “attacks” filled with lies,distortions and inflammatory images.
This summer, a group of well-financed conservative activists had an idea for what they hoped would be a last-minute game changer in the presidential race A new anti-Obama DVD is dropping into voters’ mailboxes, claiming that the president is the love child of an illicit relationship between his mother, Stanley Ann Dunham, and Frank Marshall Davis, a Communist party loyalist. They went to the unusual length of arranging a focus group to test anti-Obama films. Conducted by Frank Luntz, the well-known Republican research analyst, a 30-person focus group looked at three choices: Dinesh D’Souza’s “2016: Obama’s America,” which theorizes that the president’s political beliefs were shaped by the radical “anticolonial” views of his Kenyan father; “The Hope and the Change,” a softer critique of the president that features interviews with disaffected former Obama supporters; and “Dreams From My Real Father,” which posits the implausible theory that the president’s real father is Mr. Davis, and that Mr. Davis indoctrinated him with Marxist views early on.
James N. DruckmanHow does television affect politicalbehavior?Half the participants watched a televised version of thefirst Kennedy-Nixon debateThe other half listened to an audio version.I used this debate in part because despite popularconceptions, there is no extant evidence that televisionimages had any impact on audience reactions. I find thattelevision images have significant effects—they: affect overall debate evaluations prime people to rely more on personality perceptions in their evaluations enhance what people learnTelevision images matter in politics, and may haveindeed played an important role in the first Kennedy-Nixon debate.Note: only used students who had not heard about theKennedy-Nixon debate.
Experiments are social situations that are constructed to simulate important features of our daily lives. They involve: Random Assignment of Participants (subjects randomly assigned to diff. conditions) Standardization (Control, Manipulating variables) Placebo effects (control condition) Well-designed experiments also guard against Experimental bias (1) Expectancy effects (2) Experimenter bias (3) Demand characteristics (use of deception)
Standardization (Control: Manipulating variables): goal is to vary just one or two factors (independent variables) at a time to pinpoint how changes in just one or two things influence behavior (the dependent variable). The experimental conditions are exactly the same in every respect, except one factor, the independent variable. Thus, if the dependent variable varies across the experimental conditions, we know the manipulation caused the change.
Random assignment: Because we can never include (and statistically control for) all the factors that might influence the dependent variable, randomly assigning each subject to a treatment group eliminates all extraneous factors in one fell swoop. Because the people in each treatment group would—in every conceivable way—average about the same for most characteristics.
Same design for clinical trials to determine causal impact of treatments for alcoholism, Hepatitis C, etc.Why do funding agencies in health insist on experiments to test impact of treatments?
Placebo effects: Control conditions help determine the extent of placebo effects (simply being in an experiment) in an experimental manipulation. Not all experiments have control conditions, but the ones that do allow one to rule out placebo effects.
Expectancy effects, Experimenter bias, Demand characteristics. Use deception to ensure that subjects cannot determine the relevant hypotheses or tests. Example: impact of real campaign ads
Replication: “You gotta problem?”—with something about the experiment (the treatments, setting, subjects)? Experimenters have a solution: replicate the experiment on every dimension except the one you want to vary. If the results are different, it is likely due to that one factor. If they are the same, you have more confidence in the results. Example: make the “attack” ads more negative, or have people watch more ads over time, or use subjects from another city, etc.
Threats to external validity: David Sears on college sophomores. For what types of attitudes and behaviors does Sears say findings from college sophomores may not generalize to people in general? ▪ They have less crystallized attitudes, easier to change ▪ They are very attentive and willing to be effortful and complete any task ▪ They tend to be more socially liberal than the general population Responding to Sears: Studies show that as long as there is some variation on the independent variable among student subjects, experimental effects can still be detected. Only when there is no variation will we fail to detect the impact of the independent variable on the dependent variable. Even if the content of people’s attitudes & behavior varies across subgroups and cultures, the process by which they think and act does not vary. Bottom line: critics are the ones who need to defend their criticisms of using college sophomores. Blanket dismissals won’t cut it!
Advantages (internal validity) 1. Ability to test causal inferences. 2. Ability to explore the details of process. 3. Relative economy. Disadvantages (external validity) 1. Sampling bias: Unrepresentative subject pools (i.e., over-use of college sophomores). 2. Artificial environment, esp. in terms of “Mundane realism” (any superficial appearance of reality. The similarity of experimental events to everyday experiences.) ▪ But need to consider “Experimental realism” (the experiment captures the intended essence of the situation when subjects are caught up in the experiment and are truly influenced by it). 3. In the treatment condition, everyone is exposed to the treatment which may not be an accurate portrayal of reality (e.g., attention levels, remote controls).
Survey & experiments (influence of TV violence, cigarette smoking) Survey experiments (hybrid) ▪ Question wording experiments (text only) ▪ Internet surveys (video experiments via survey)
In 1991, big correlation between whites’ negative stereotypes of blacks and their opposition to welfare. Most whites mistakenly assumed that blacks were the principal beneficiary of welfare.Can we conclude that negative racial stereotypes cause opposition to welfare? Why or why not?
Now think about a [black woman, white woman] in her early twenties with a ten year old child and who has been on welfare for the past year. ½ respondents randomly assigned to the black welfare mother condition Other ½ respondents randomly assigned to the white welfare mother condition Question: why not ask everyone about both the black and the white welfare mothers and then compare responses? The two conditions (questions) are exactly the same in every respect, except for the race of the welfare mother. Any differences in responses must be due to the manipulation.
Conclusion: The much higher correlation between attitudes toward Black than White welfare motherand opposition to welfare shows: When Whites think of welfare, they think of Black welfaremothers, not white welfare mothers.
Appendix I. Morphed ImagesFor each candidate, 3 images were morphed.1 This procedure removes distinctive features and creates a very“average” face; it also increases attractiveness. Because each candidate shares 2 images, the resulting faces areextremely similar while appearing to be distinct individuals. This methodology diminishes the likelihood thatsupport differentials across treatment groups are due to different target candidates.White Candidate 1: + +A B CWhite Candidate 2: + +A B D
Light-skinned Black Candidate: + +A E F Dark-skinned Black Candidate1: + +E F G The dark-black candidate was purposefully generated from all blackimages so that he is different from the light-skinned black candidate in bothcomplexion and phenotype (his nose and lips are more afrocentric). Thehair and eyebrows of the light-skinned candidate have also been digitallylightened1 .