Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Weighting online data
Jeffrey Henning
Executive Director
Market Research Institute International
August
2019
Sponsors
Communication
Gold
Silver
Visit NewMR.org
www.marketresearchcourses.org
2
Learning Objective:
Describe the challenges in obtaining
representative samples and how
representative samples can be
impr...
• Post-stratification weighting is viewed as a
common solution to removing sampling bias.
• But it is often misrepresented...
U.S. Men Women
18 to 54
79,184,164
169 responses
469K weight
79,017,200
199 responses
397K weight
55+
36,301,576
15 respon...
• Do you want to set minimum and maximum
weights? If so, to what? Why?
• How will you simplify the weighting scheme
if sam...
Age
Sex
Region
Race/ethnicity
Education
level
Household
income
Proprietary
measure
Rim Weighting / Raking
Age
Age by sex
Age by race/ethnicity
Age by education
Age by region
Sex
Sex by race/ethnicity
Sex by education
Sex by regi...
Age Age by sex
Age by
race/ethnicity
Age by
education
Age by region
Sex
Sex by
race/ethnicity
Sex by
education
Sex by regi...
• “No religious or political questions.”
• “You were all over the place. Starts by asking
questions on products and servic...
Pew Weights
Probability Panels
Differently than it does
Opt-in Panels
• Implicit assumption is respondents in a
demographic group are representative of the
people in that group we did not surv...
61%
74%
84%
89%
81%
65%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010...
• Some researchers weight convenience
samples...
§ In the hope it does no harm
§ In the belief it improves quality
§ For t...
• Demographic weighted surveys - Reflect the composition of the target
audience, often using cells comprised of age, gende...
Pew Research into Weighting
Wait, Wait
Waiting until the weighting stage
to adjust is too late. The
combination of coverage error and
nonresponse in o...
Men Women
18 to 54 79,184,164
169 133 responses
595K weight
79,017,200
199 132 responses
599K weight
55+ 36,301,576
15 61 ...
Vendor A
• Sex by age by
region
Vendor B
• Sex by age
• Region
Vendor C
• Sex
• Age
• Education
• Census region
• Race/eth...
• For non-probability samples not using quota
sampling:
§ Don’t weight the results
• For non-probability samples using quo...
• Include disqualified respondents when
screening general population
§ Screener should contain weighting variables
• If ca...
Call for Further Research
• Analysis of different
weighting schemes,
especially political
questions and religion
for busin...
For Further Reading
Jeffrey Henning
Executive Director
Market Research Institute International
jhenning@mrii.org
@jhenning
https://www.linkedi...
Sponsors
Communication
Gold
Silver
Visit NewMR.org
August
2019
Q & A
Ray Poynter
NewMR
Jeffrey Henning
MRII
(Market Research
Institute International)
How to weight online data
Upcoming SlideShare
Loading in …5
×

How to weight online data

43 views

Published on

The industry thinking on weighting nonprobability surveys is changing. Most online survey software doesn’t offer weighting, and traditionally the advice has been that weighting nonprobability surveys doesn’t improve the results and can even skew the findings. A key reason not to weight is that nonresponse bias from key demographics can’t be corrected through weighting: for instance, senior citizens are underrepresented in online surveys and those who do participate differ significantly from non-responders in health, education, and engagement with technology.

Yet weighting of nonprobability surveys has been on the rise among public opinion researchers and deserves broader consideration among market researchers. Four key reasons that researchers should consider weighting:

- Weighting, done right, improves the representativeness of the results.
- Weighting forces the demographics to more closely match the market, reducing client anxiety.
- Weighting compensates for incomplete quota cells and can minimise the effects of unintentionally interlocking quotas.
- Weighting produces meaningful toplines in surveys with deliberate oversamples.

Jeffrey Henning, PRC will spell out how and when to weight your online surveys, with appropriate caveats and advice for when not to weight such surveys.
This presentation was part of the 'Challenges in Modern Market Research' webinar on 29 August 2019.

Published in: Education
  • Be the first to comment

  • Be the first to like this

How to weight online data

  1. 1. Weighting online data Jeffrey Henning Executive Director Market Research Institute International August 2019
  2. 2. Sponsors Communication Gold Silver Visit NewMR.org
  3. 3. www.marketresearchcourses.org 2
  4. 4. Learning Objective: Describe the challenges in obtaining representative samples and how representative samples can be improved at the selection stage or through weighting. Challenges: Traditionally we have recommended weighting for probability samples. Should we recommend weighting for non-probability samples? When?
  5. 5. • Post-stratification weighting is viewed as a common solution to removing sampling bias. • But it is often misrepresented as a simple process of arithmetic… Weighting
  6. 6. U.S. Men Women 18 to 54 79,184,164 169 responses 469K weight 79,017,200 199 responses 397K weight 55+ 36,301,576 15 responses 2,420K weight 43,154,705 17 responses 2,539K weight Cell Weighting
  7. 7. • Do you want to set minimum and maximum weights? If so, to what? Why? • How will you simplify the weighting scheme if sample balance is too low (e.g., <70%)? • Which questions to weight on? Editorial Judgments
  8. 8. Age Sex Region Race/ethnicity Education level Household income Proprietary measure Rim Weighting / Raking
  9. 9. Age Age by sex Age by race/ethnicity Age by education Age by region Sex Sex by race/ethnicity Sex by education Sex by region Race/ethnicity Race/ethnicity by education Race/ethnicity by region Education Education by region Census division Political party affiliation Political ideology Voter registration Evangelical Christian identification Raking with Interlocked Variables…
  10. 10. Age Age by sex Age by race/ethnicity Age by education Age by region Sex Sex by race/ethnicity Sex by education Sex by region Race/ethnicity Race/ethnicity by education Race/ethnicity by region Education Education by region Census division Political party affiliation Political ideology Voter registration Evangelical Christian identification …Using these Interlocked Variables
  11. 11. • “No religious or political questions.” • “You were all over the place. Starts by asking questions on products and services and ends on political preferences.” • “Do not ask political questions. It seemed very strange at the end.” Complaints about Weighting Questions Political party affiliation Political ideology Voter registration Evangelical Christian identification
  12. 12. Pew Weights Probability Panels Differently than it does Opt-in Panels
  13. 13. • Implicit assumption is respondents in a demographic group are representative of the people in that group we did not survey § Not true for seniors • Non-Internet seniors differ on many dimensions from Internet-using seniors § Not true for immigrant communities • Acculturated vs. unacculturated Key Assumption Behind Weighting
  14. 14. 61% 74% 84% 89% 81% 65% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 % of U.S. adults with Internet Access U.S. adults Less than $30,000 Less than high school graduate 65+ Internet Surveys Decimate the Population Source: Pew Research
  15. 15. • Some researchers weight convenience samples... § In the hope it does no harm § In the belief it improves quality § For the fact it redistributes demographics to match target populations Assumptions Behind Weighting
  16. 16. • Demographic weighted surveys - Reflect the composition of the target audience, often using cells comprised of age, gender and region. However, David Yeager and Jon Krosnick determined that demographically weighting non- probability Internet samples to known population values did not consistently produce more representative results. • Demographic and attitudinal weighted surveys - Some have also used attitudinal questions in their weighting functions, but there has been no academic validation of this. • Propensity weighted surveys - Propensity score weighting adjusts for the likelihood of respondents to be online based on their demographics. • Non-parametric weighted surveys - Brian Fine of ORU demonstrated that CART analysis could be used to model representative results by modeling the dependent variable as panelist source. Not yet been independently validated. Weighting Recommendations (2014)
  17. 17. Pew Research into Weighting
  18. 18. Wait, Wait Waiting until the weighting stage to adjust is too late. The combination of coverage error and nonresponse in online panels generally creates a sample that is beyond fixing post hoc. We need to do more at the selection stage. Reg Baker (2013) ESOMAR Ambassador
  19. 19. Men Women 18 to 54 79,184,164 169 133 responses 595K weight 79,017,200 199 132 responses 599K weight 55+ 36,301,576 15 61 responses 595K weight 43,154,705 17 72 responses 599K weight Quota Sampling
  20. 20. Vendor A • Sex by age by region Vendor B • Sex by age • Region Vendor C • Sex • Age • Education • Census region • Race/ethnicity • Population density Vendor D • Sex by age • Sex by education • Age by education • Census region • Race/ethnicity Common Quota Schemes
  21. 21. • For non-probability samples not using quota sampling: § Don’t weight the results • For non-probability samples using quota sampling: § Weight to correct oversamples § Weight interim studies using quota sampling to correct for slow-filling cells § May not be able to weight crosstabs (technical limitation of many survey packages) Recommendations
  22. 22. • Include disqualified respondents when screening general population § Screener should contain weighting variables • If can’t include disqualified respondents, run omnibus questions to estimate population totals • Note that it can be time consuming to track down updated benchmarking information; design your questions based on the benchmarks you find Easily Overlooked Items
  23. 23. Call for Further Research • Analysis of different weighting schemes, especially political questions and religion for business questions • Research into weight trimming • Research into sample balance • Research into N < 2000 studies
  24. 24. For Further Reading
  25. 25. Jeffrey Henning Executive Director Market Research Institute International jhenning@mrii.org @jhenning https://www.linkedin.com/in/jhenning/
  26. 26. Sponsors Communication Gold Silver Visit NewMR.org
  27. 27. August 2019 Q & A Ray Poynter NewMR Jeffrey Henning MRII (Market Research Institute International)

×