The industry thinking on weighting nonprobability surveys is changing. Most online survey software doesn’t offer weighting, and traditionally the advice has been that weighting nonprobability surveys doesn’t improve the results and can even skew the findings. A key reason not to weight is that nonresponse bias from key demographics can’t be corrected through weighting: for instance, senior citizens are underrepresented in online surveys and those who do participate differ significantly from non-responders in health, education, and engagement with technology.
Yet weighting of nonprobability surveys has been on the rise among public opinion researchers and deserves broader consideration among market researchers. Four key reasons that researchers should consider weighting:
- Weighting, done right, improves the representativeness of the results.
- Weighting forces the demographics to more closely match the market, reducing client anxiety.
- Weighting compensates for incomplete quota cells and can minimise the effects of unintentionally interlocking quotas.
- Weighting produces meaningful toplines in surveys with deliberate oversamples.
Jeffrey Henning, PRC will spell out how and when to weight your online surveys, with appropriate caveats and advice for when not to weight such surveys.
This presentation was part of the 'Challenges in Modern Market Research' webinar on 29 August 2019.
Describe the challenges in obtaining
representative samples and how
representative samples can be
improved at the selection stage or
Traditionally we have recommended
weighting for probability samples.
Should we recommend weighting for
non-probability samples? When?
• Post-stratification weighting is viewed as a
common solution to removing sampling bias.
• But it is often misrepresented as a simple
process of arithmetic…
U.S. Men Women
18 to 54
• Do you want to set minimum and maximum
weights? If so, to what? Why?
• How will you simplify the weighting scheme
if sample balance is too low (e.g., <70%)?
• Which questions to weight on?
Rim Weighting / Raking
Age by sex
Age by race/ethnicity
Age by education
Age by region
Sex by race/ethnicity
Sex by education
Sex by region
Education by region
Raking with Interlocked Variables…
Age Age by sex
Age by region
Sex by region Race/ethnicity
…Using these Interlocked Variables
• “No religious or political questions.”
• “You were all over the place. Starts by asking
questions on products and services and ends on
• “Do not ask political questions. It seemed very
strange at the end.”
Complaints about Weighting Questions
Differently than it does
• Implicit assumption is respondents in a
demographic group are representative of the
people in that group we did not survey
§ Not true for seniors
• Non-Internet seniors differ on many dimensions
from Internet-using seniors
§ Not true for immigrant communities
• Acculturated vs. unacculturated
Key Assumption Behind Weighting
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
% of U.S. adults with Internet Access
U.S. adults Less than $30,000 Less than high school graduate 65+
Internet Surveys Decimate the Population
Source: Pew Research
• Some researchers weight convenience
§ In the hope it does no harm
§ In the belief it improves quality
§ For the fact it redistributes demographics to
match target populations
Assumptions Behind Weighting
• Demographic weighted surveys - Reflect the composition of the target
audience, often using cells comprised of age, gender and region. However,
David Yeager and Jon Krosnick determined that demographically weighting non-
probability Internet samples to known population values did not consistently
produce more representative results.
• Demographic and attitudinal weighted surveys - Some have also used
attitudinal questions in their weighting functions, but there has been no
academic validation of this.
• Propensity weighted surveys - Propensity score weighting adjusts for the
likelihood of respondents to be online based on their demographics.
• Non-parametric weighted surveys - Brian Fine of ORU demonstrated that CART
analysis could be used to model representative results by modeling the
dependent variable as panelist source. Not yet been independently validated.
Weighting Recommendations (2014)
Waiting until the weighting stage
to adjust is too late. The
combination of coverage error and
nonresponse in online panels
generally creates a sample that is
beyond fixing post hoc. We need
to do more at the selection stage.
Reg Baker (2013)
• Sex by age by
• Sex by age
• Census region
• Sex by age
• Sex by
• Age by
• Census region
Common Quota Schemes
• For non-probability samples not using quota
§ Don’t weight the results
• For non-probability samples using quota
§ Weight to correct oversamples
§ Weight interim studies using quota sampling to
correct for slow-filling cells
§ May not be able to weight crosstabs (technical
limitation of many survey packages)
• Include disqualified respondents when
screening general population
§ Screener should contain weighting variables
• If can’t include disqualified respondents, run
omnibus questions to estimate population
• Note that it can be time consuming to track
down updated benchmarking information;
design your questions based on the
benchmarks you find
Easily Overlooked Items
Call for Further Research
• Analysis of different
questions and religion
for business questions
• Research into weight
• Research into sample
• Research into N < 2000