Reducing Bias in Public Opinion Polls

2,311 views
2,144 views

Published on

Reducing Bias in Public Opinion Polls - By Autumn Carter
Presentation Delivered at Stanford University, Summer Research College, August 2009

Presentation of some of my results from my research. Over the course of the project, I analyzed 7 survey experiments across 9 unique vendors, who conducted the survey via phone or the internet and using either a probability sample or a non-probability sample. In all, about 10000 unique individuals were surveyed.

This presentation highlights the question bias of Agree/Disagree type questions and the response bias associated with the Acquiescence Effect.

I conducted my research through Stanford\'s Political Psychology Research Group. pprg.stanford.edu

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,311
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Reducing Bias in Public Opinion Polls

  1. 1. Reducing Bias in Public Opinion Polls Autumn Carter Professor Jon Krosnick Summer Research College 2009
  2. 2. Satisfy Your Constituents... Or Suffer the Consequences.
  3. 3. So, how do we effectively determine what Americans are feeling?
  4. 4. Public Opinion Polling
  5. 5. But Are the Polls Accurate? ? =
  6. 6. Critics Don't Think So... “ The dirty little secret of the polling industry is that, all too often, its findings are based on flawed methodology and dubious assumptions.” -Arianna Huffington, Huffington Post Founder
  7. 7. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree Disagree
  8. 8. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree: 59.6% Disagree: 40.4%
  9. 9. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree: 59.6% Disagree: 40.4% Form 2 Social conditions are more to blame than individuals for crime and lawlessness in this country. Agree Disagree
  10. 10. Sample Public Opinion Question Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree: 59.6% Disagree: 40.4% Form 2 Social conditions are more to blame than individuals for crime and lawlessness in this country. Agree: 56.8% Disagree: 43.2% Acquiescence Effect: 16.4%
  11. 11. What is Acquiescence? “ ...a presumed tendency for respondents to agree with attitude statements presented to them.” -Schuman and Presser (1981)
  12. 12. Why do Respondents Acquiesce? <ul><li>Norms of Conduct : Be polite and agreeable </li></ul><ul><li>Status Differential : Defer to higher status </li></ul><ul><li>Satisficing : Choosing the easiest response because it requires less thinking </li></ul>
  13. 13. Can we reduce acquiescence by administering the survey via the Internet instead of by telephone?
  14. 14. Would Internet Administration Reduce Acquiescence? <ul><li>Norms of Conduct : Be polite and agreeable </li></ul><ul><ul><ul><li>Do you have to be polite to a computer? </li></ul></ul></ul><ul><li>Status Differential : Defer to higher status </li></ul><ul><ul><ul><li>Is there a status difference without a human interviewer? </li></ul></ul></ul><ul><li>Satisficing : Choosing the easiest response because it requires less thinking </li></ul><ul><ul><ul><li>Are people less likely to satisfice on Internet surveys? (Chang & Krosnick, in press) </li></ul></ul></ul>
  15. 15. The Present Research <ul><li>Replicate Schuman & Presser (1981) </li></ul><ul><li>Mode effects: Telephone vs. the Internet </li></ul><ul><li>Sample effects: Representative vs. Non-Representative </li></ul>
  16. 16. How Do You Survey a Representative Sample Via the Internet? <ul><li>Knowledge Networks randomly dialed phone numbers and invited people to join a panel (of about 50,000 adults). </li></ul><ul><li>Provided Internet access to homes without it. </li></ul><ul><li>Periodically emails about 1,000 panelists to complete surveys and receive incentives. </li></ul>
  17. 17. A Classic Acquiescence Experiment Form 1 Individuals are more to blame than social conditions for crime and lawlessness in this country. Agree Disagree Form 2 Social conditions are more to blame than individuals for crime and lawlessness in this country. Agree Disagree
  18. 18. Replication in a National Survey Telephone survey of a representative sample: Internet survey of a representative sample: * p < 0.05 + p <0.10 11.5%* 7.1%+ TelephoneXInternet Interaction: n.s.
  19. 19. What about Internet surveys of people who are not representative of the national population?
  20. 20. Non-Probability Sample Internet Surveys <ul><li>Volunteers click on banner ads and “opt-in” into the panel. </li></ul>
  21. 21. You've all seen these...
  22. 22. Non-Probability Sample Internet Surveys <ul><li>Volunteers click on banner ads and “opt-in” to the panel. </li></ul><ul><li>Participants receive money or prize. </li></ul><ul><li>Perhaps they acquiescence less because they like taking surveys and have more practice. </li></ul><ul><li>In fact, previous research has found less acquiescence vs. a telephone survey (Chang and Krosnick, in press) </li></ul>
  23. 23. Acquiescence Bias * * * * * † * † † *
  24. 24. If anything, non-probability samples increase acquiescence. So we should be cautious when interpreting these types of surveys.
  25. 25. What if we just didn't ask questions that ask respondents to agree or disagree with a statement?
  26. 26. Balanced Forced-Choice Questions Form 1 Which do you think is more responsible for crime and lawlessness in this country: individuals or social conditions ? Individuals Social Conditions Form 2 Which do you think is more responsible for crime and lawlessness in this country: social conditions or individuals ? Social Conditions Individuals
  27. 27. Response Order Effects ns ns ns ns ns ns ns ns ns
  28. 28. Conclusions <ul><li>Acquiescence bias is real and can undermine public opinion polls and possibly the democratic process. </li></ul><ul><li>Internet administration did not significantly reduce acquiescence. </li></ul><ul><li>Non-probability sample surveys showed, if anything, more acquiescence. </li></ul><ul><li>BUT WAIT! You can simply ask the questions using the balanced forced choice format, eliminating order effects. </li></ul>

×