The industry thinking on weighting nonprobability surveys is changing. Most online survey software doesn’t offer weighting, and traditionally the advice has been that weighting nonprobability surveys doesn’t improve the results and can even skew the findings. A key reason not to weight is that nonresponse bias from key demographics can’t be corrected through weighting: for instance, senior citizens are underrepresented in online surveys and those who do participate differ significantly from non-responders in health, education, and engagement with technology. Yet weighting of nonprobability surveys has been on the rise among public opinion researchers and deserves broader consideration among market researchers. Four key reasons that researchers should consider weighting: - Weighting, done right, improves the representativeness of the results. - Weighting forces the demographics to more closely match the market, reducing client anxiety. - Weighting compensates for incomplete quota cells and can minimise the effects of unintentionally interlocking quotas. - Weighting produces meaningful toplines in surveys with deliberate oversamples. Jeffrey Henning, PRC will spell out how and when to weight your online surveys, with appropriate caveats and advice for when not to weight such surveys. This presentation was part of the 'Challenges in Modern Market Research' webinar on 29 August 2019.

1. Weighting online data
Jeffrey Henning
Executive Director
Market Research Institute International
August
2019

2. Sponsors
Communication
Gold
Silver
Visit NewMR.org

3. www.marketresearchcourses.org
2

4. Learning Objective:
Describe the challenges in obtaining
representative samples and how
representative samples can be
improved at the selection stage or
through weighting.
Challenges:
Traditionally we have recommended
weighting for probability samples.
Should we recommend weighting for
non-probability samples? When?

5. • Post-stratification weighting is viewed as a
common solution to removing sampling bias.
• But it is often misrepresented as a simple
process of arithmetic…
Weighting

6. U.S. Men Women
18 to 54
79,184,164
169 responses
469K weight
79,017,200
199 responses
397K weight
55+
36,301,576
15 responses
2,420K weight
43,154,705
17 responses
2,539K weight
Cell Weighting

7. • Do you want to set minimum and maximum
weights? If so, to what? Why?
• How will you simplify the weighting scheme
if sample balance is too low (e.g., <70%)?
• Which questions to weight on?
Editorial Judgments

8. Age
Sex
Region
Race/ethnicity
Education
level
Household
income
Proprietary
measure
Rim Weighting / Raking

9. Age
Age by sex
Age by race/ethnicity
Age by education
Age by region
Sex
Sex by race/ethnicity
Sex by education
Sex by region
Race/ethnicity
Race/ethnicity by
education
Race/ethnicity by
region
Education
Education by region
Census division
Political party
affiliation
Political ideology
Voter registration
Evangelical Christian
identification
Raking with Interlocked Variables…

10. Age Age by sex
Age by
race/ethnicity
Age by
education
Age by region
Sex
Sex by
race/ethnicity
Sex by
education
Sex by region Race/ethnicity
Race/ethnicity
by education
Race/ethnicity
by region
Education
Education by
region
Census
division
Political party
affiliation
Political
ideology
Voter
registration
Evangelical
Christian
identification
…Using these Interlocked Variables

11. • “No religious or political questions.”
• “You were all over the place. Starts by asking
questions on products and services and ends on
political preferences.”
• “Do not ask political questions. It seemed very
strange at the end.”
Complaints about Weighting Questions
Political party
affiliation
Political
ideology
Voter
registration
Evangelical
Christian
identification

12. Pew Weights
Probability Panels
Differently than it does
Opt-in Panels

13. • Implicit assumption is respondents in a
demographic group are representative of the
people in that group we did not survey
§ Not true for seniors
• Non-Internet seniors differ on many dimensions
from Internet-using seniors
§ Not true for immigrant communities
• Acculturated vs. unacculturated
Key Assumption Behind Weighting

14. 61%
74%
84%
89%
81%
65%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
% of U.S. adults with Internet Access
U.S. adults Less than $30,000 Less than high school graduate 65+
Internet Surveys Decimate the Population
Source: Pew Research

15. • Some researchers weight convenience
samples...
§ In the hope it does no harm
§ In the belief it improves quality
§ For the fact it redistributes demographics to
match target populations
Assumptions Behind Weighting

16. • Demographic weighted surveys - Reflect the composition of the target
audience, often using cells comprised of age, gender and region. However,
David Yeager and Jon Krosnick determined that demographically weighting non-
probability Internet samples to known population values did not consistently
produce more representative results.
• Demographic and attitudinal weighted surveys - Some have also used
attitudinal questions in their weighting functions, but there has been no
academic validation of this.
• Propensity weighted surveys - Propensity score weighting adjusts for the
likelihood of respondents to be online based on their demographics.
• Non-parametric weighted surveys - Brian Fine of ORU demonstrated that CART
analysis could be used to model representative results by modeling the
dependent variable as panelist source. Not yet been independently validated.
Weighting Recommendations (2014)

17. Pew Research into Weighting

19. Wait, Wait
Waiting until the weighting stage
to adjust is too late. The
combination of coverage error and
nonresponse in online panels
generally creates a sample that is
beyond fixing post hoc. We need
to do more at the selection stage.
Reg Baker (2013)
ESOMAR Ambassador

20. Men Women
18 to 54 79,184,164
169 133 responses
595K weight
79,017,200
199 132 responses
599K weight
55+ 36,301,576
15 61 responses
595K weight
43,154,705
17 72 responses
599K weight
Quota Sampling

21. Vendor A
• Sex by age by
region
Vendor B
• Sex by age
• Region
Vendor C
• Sex
• Age
• Education
• Census region
• Race/ethnicity
• Population
density
Vendor D
• Sex by age
• Sex by
education
• Age by
education
• Census region
• Race/ethnicity
Common Quota Schemes

22. • For non-probability samples not using quota
sampling:
§ Don’t weight the results
• For non-probability samples using quota
sampling:
§ Weight to correct oversamples
§ Weight interim studies using quota sampling to
correct for slow-filling cells
§ May not be able to weight crosstabs (technical
limitation of many survey packages)
Recommendations

23. • Include disqualified respondents when
screening general population
§ Screener should contain weighting variables
• If can’t include disqualified respondents, run
omnibus questions to estimate population
totals
• Note that it can be time consuming to track
down updated benchmarking information;
design your questions based on the
benchmarks you find
Easily Overlooked Items

24. Call for Further Research
• Analysis of different
weighting schemes,
especially political
questions and religion
for business questions
• Research into weight
trimming
• Research into sample
balance
• Research into N < 2000
studies

25. For Further Reading

26. Jeffrey Henning
Executive Director
Market Research Institute International
jhenning@mrii.org
@jhenning
https://www.linkedin.com/in/jhenning/

27. Sponsors
Communication
Gold
Silver
Visit NewMR.org

28. August
2019
Q & A
Ray Poynter
NewMR
Jeffrey Henning
MRII
(Market Research
Institute International)