This document discusses different approaches to weighting a probability-based online panel with multiple waves of recruitment. It summarizes the evolution of recruitment for the Life in Australia panel from 2016-2019, including adding new respondents through mobile phone RDD and an area-based sample from the ABS. The document evaluates different weighting approaches, finding a "super-population model" that predicts values for non-sampled units using auxiliary data has the highest weighting efficiency and lowest bias compared to design-based weights or propensity modeling. It concludes a model-based approach is simpler, more efficient and universally applicable for a long-term panel with multiple refreshes.
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Dina Neiger - Chief Statistician (Social Research Centre)
A solutions-based approach, illustrated by case studies, which show how inferences can be improved from surveys administered to biased, low response rate and non-probability samples.
It addresses how to improve the accuracy of the survey estimates we generate from poorer quality and non-probability samples.
Workshop session 4 - Optimal sample designs for general community telephone s...The Social Research Centre
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Dina Neiger - Chief Statistician (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Dina Neiger - Chief Statistician (Social Research Centre)
A solutions-based approach, illustrated by case studies, which show how inferences can be improved from surveys administered to biased, low response rate and non-probability samples.
It addresses how to improve the accuracy of the survey estimates we generate from poorer quality and non-probability samples.
Workshop session 4 - Optimal sample designs for general community telephone s...The Social Research Centre
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Dina Neiger - Chief Statistician (Social Research Centre)
In this study, the effect of combining variables from the different data sources for student academic performance prediction was examined using three state-of-the–art classifiers: Decision Tree (DT), Artificial Neural Network (ANN) and Support Vector Machine (SVM). The study examined the use of heterogeneous multi-model ensemble techniques to predict student academic performance based on the combination of these classifiers and three different data sources. A quantitative approach was used to develop the various base classifier models while the ensemble models were developed using stacked generalisation ensemble method in order to overcome the individual weaknesses of the different models. Variables were extracted from the institution’s Student Record System and Learning Management System (Moodle) and from a structured student questionnaire. At present, negligible work has been done using this integrated approach and ensemble techniques especially with aggregated learner data in performance prediction in HE. The empirical results obtained show that the ensemble models.........................
sampling design methods in research it discuses about what is research, types of research and types of sampling design it considering about probability and non-probability sampling
Workshop session 10 - Alternatives to CATI (3) address-based sampling and pus...The Social Research Centre
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Anna Lethborg - Research Director (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Ben Phillips, Senior Research Director, Survey Methodology (Social Research Centre)
The purpose of this paper is to explore the development of universal design and to consider its application to large scale assessments. Building on universal design principles presented by the Center for Universal Design, seven elements of universally designed assessments are identified and described in this paper. The seven elements are:
1. Inclusive assessment population
2. Precisely defined constructs
3. Accessible, non-biased items
4. Amendable to accommodations
5. Simple, clear, and intuitive instructions and procedures
6. Maximum readability and comprehensibility
7. Maximum legibility
Each of the elements is explored in this paper.
Evaluation of Library STEM Programs: Learning from the BISE ProjectNCIL - STAR_Net
Teasdale R. M., Grack Nelson A. (2015)
Evaluation of Library STEM Programs: Learning from the BISE Project
University of Illinois at Urbana-Champaign, Science Museum of Minnesota
In this talk we discuss the results of the survey of software ecosystems researchers conducted in October-December 2014. Researchers have been asked to identify the current trends in ecosystems’ research as well as the challenges the research community has to address in the coming years. We augment discussion of the trends identified by the community by the review of some of the recent results on software ecosystems.
Joint work with Tom Mens.
Transferring biodiversity models for conservation: Opportunities and challengesPhil Bouchet
Presentation at the 2018 annual conference of the British Ecological Society, Birmingham, UK. Based on following publication: Sequeira AMM, Bouchet PJ, Yates KL, Mengersen K, Caley MJ. 2018. Transferring biodiversity models for conservation: Opportunities and challenges. Methods in Ecology and Evolution, 9(5): 1250-1264.
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Presentation by Paul J. Lavrakas, PhD - Senior Methodological Adviser (Social Research Centre)
More Related Content
Similar to Weighting a probability online panel with multiple waves of recruitment
In this study, the effect of combining variables from the different data sources for student academic performance prediction was examined using three state-of-the–art classifiers: Decision Tree (DT), Artificial Neural Network (ANN) and Support Vector Machine (SVM). The study examined the use of heterogeneous multi-model ensemble techniques to predict student academic performance based on the combination of these classifiers and three different data sources. A quantitative approach was used to develop the various base classifier models while the ensemble models were developed using stacked generalisation ensemble method in order to overcome the individual weaknesses of the different models. Variables were extracted from the institution’s Student Record System and Learning Management System (Moodle) and from a structured student questionnaire. At present, negligible work has been done using this integrated approach and ensemble techniques especially with aggregated learner data in performance prediction in HE. The empirical results obtained show that the ensemble models.........................
sampling design methods in research it discuses about what is research, types of research and types of sampling design it considering about probability and non-probability sampling
Workshop session 10 - Alternatives to CATI (3) address-based sampling and pus...The Social Research Centre
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Anna Lethborg - Research Director (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Ben Phillips, Senior Research Director, Survey Methodology (Social Research Centre)
The purpose of this paper is to explore the development of universal design and to consider its application to large scale assessments. Building on universal design principles presented by the Center for Universal Design, seven elements of universally designed assessments are identified and described in this paper. The seven elements are:
1. Inclusive assessment population
2. Precisely defined constructs
3. Accessible, non-biased items
4. Amendable to accommodations
5. Simple, clear, and intuitive instructions and procedures
6. Maximum readability and comprehensibility
7. Maximum legibility
Each of the elements is explored in this paper.
Evaluation of Library STEM Programs: Learning from the BISE ProjectNCIL - STAR_Net
Teasdale R. M., Grack Nelson A. (2015)
Evaluation of Library STEM Programs: Learning from the BISE Project
University of Illinois at Urbana-Champaign, Science Museum of Minnesota
In this talk we discuss the results of the survey of software ecosystems researchers conducted in October-December 2014. Researchers have been asked to identify the current trends in ecosystems’ research as well as the challenges the research community has to address in the coming years. We augment discussion of the trends identified by the community by the review of some of the recent results on software ecosystems.
Joint work with Tom Mens.
Transferring biodiversity models for conservation: Opportunities and challengesPhil Bouchet
Presentation at the 2018 annual conference of the British Ecological Society, Birmingham, UK. Based on following publication: Sequeira AMM, Bouchet PJ, Yates KL, Mengersen K, Caley MJ. 2018. Transferring biodiversity models for conservation: Opportunities and challenges. Methods in Ecology and Evolution, 9(5): 1250-1264.
Similar to Weighting a probability online panel with multiple waves of recruitment (20)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Presentation by Paul J. Lavrakas, PhD - Senior Methodological Adviser (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy (Social Research Centre)
Workshop session 5 - the effectiveness of standard methods to improve general...The Social Research Centre
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy and Ben Phillips - Senior Research Director, Survey Methodology (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Kane Chapman - General Manager (SamplePages)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Ben Phillips, Senior Research Director, Survey Methodology (Social Research Centre)
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy (Social Research Centre)
Increasing cooperation in telephone surveys with the progressive engagement t...The Social Research Centre
Paul Myers, Paul J Lavrakas, Darren Pennay and Natasha Vickers
American Association For Public Opinion Research 71st Annual Conference 2016, Austin, Texas
0x01 - Newton's Third Law: Static vs. Dynamic AbusersOWASP Beja
f you offer a service on the web, odds are that someone will abuse it. Be it an API, a SaaS, a PaaS, or even a static website, someone somewhere will try to figure out a way to use it to their own needs. In this talk we'll compare measures that are effective against static attackers and how to battle a dynamic attacker who adapts to your counter-measures.
About the Speaker
===============
Diogo Sousa, Engineering Manager @ Canonical
An opinionated individual with an interest in cryptography and its intersection with secure software development.
This presentation, created by Syed Faiz ul Hassan, explores the profound influence of media on public perception and behavior. It delves into the evolution of media from oral traditions to modern digital and social media platforms. Key topics include the role of media in information propagation, socialization, crisis awareness, globalization, and education. The presentation also examines media influence through agenda setting, propaganda, and manipulative techniques used by advertisers and marketers. Furthermore, it highlights the impact of surveillance enabled by media technologies on personal behavior and preferences. Through this comprehensive overview, the presentation aims to shed light on how media shapes collective consciousness and public opinion.
Acorn Recovery: Restore IT infra within minutesIP ServerOne
Introducing Acorn Recovery as a Service, a simple, fast, and secure managed disaster recovery (DRaaS) by IP ServerOne. A DR solution that helps restore your IT infra within minutes.
This presentation by Morris Kleiner (University of Minnesota), was made during the discussion “Competition and Regulation in Professions and Occupations” held at the Working Party No. 2 on Competition and Regulation on 10 June 2024. More papers and presentations on the topic can be found out at oe.cd/crps.
This presentation was uploaded with the author’s consent.
Sharpen existing tools or get a new toolbox? Contemporary cluster initiatives...Orkestra
UIIN Conference, Madrid, 27-29 May 2024
James Wilson, Orkestra and Deusto Business School
Emily Wise, Lund University
Madeline Smith, The Glasgow School of Art
Have you ever wondered how search works while visiting an e-commerce site, internal website, or searching through other types of online resources? Look no further than this informative session on the ways that taxonomies help end-users navigate the internet! Hear from taxonomists and other information professionals who have first-hand experience creating and working with taxonomies that aid in navigation, search, and discovery across a range of disciplines.
2. www.srcentre.com.au
Authors
Dina Neiger
Chief Statistician, the Social Research Centre
Campus Visitor, ANU Centre for Social Research and Methods
Andrew Ward
Principal Statistician, the Social Research Centre
Jack Barton
Analyst, Statistical and Survey Methods, the Social Research Centre
The Social Research Centre is a for-profit wholly owned subsidiary of the
Australian National University
2
3. www.srcentre.com.au
Acknowledgements
Darren Pennay
Founder & Senior Adviser, the Social Research Centre
Campus Visitor, ANU Centre for Social Research and Methods
Adjunct Professor, Institute for Social Science Research, the University of Queensland
Ben Phillips
Chief Survey Methodologist, the Social Research Centre
Campus Visitor, ANU Centre for Social Research and Methods
Paul J. Lavrakas
Senior Methodological Adviser, the Social Research Centre
Charles Dove
Research Consultant, the Social Research Centre
Panel Manager, Life in Australia™
3
5. www.srcentre.com.au
Australia’s only probability-based online panel
Created by the Social Research Centre in 2016
Ø Non-probability panels in Australia similar to US/Europe, relative to
probability samples, they are
o more biased
o more variable; and
o bias may be increased via traditional weighting approaches.
Ø Quality online social research
Self-funded
Ø Designed for 2,000 completed surveys per wave
Ø From 2019, expanded to provide 3,000 or more completes for a full wave
Ø Primarily split-panel waves with different clients
5
6. www.srcentre.com.au
Substantive areas measured
6
Australia’s place in
the world
Crime and justice Attitudes to
disability, autism
Political views Sun protection
attitudes /
behaviours
Social cohesion Communications
use
Awareness of
alcohol harms
Image-based abuse Cyber crime
Data privacy and
linkage
Attitudes about
immigration
Fertility Gambling Elections
Clients to date: government, academic and not-for-profit
7. www.srcentre.com.au
Operating model
Waves most months
Ø Typically whole-of-panel with occasional sub-panel selections
Ø Main client for most waves
Ø Add-on omnibus modules for secondary clients
Incentives
Ø 10 AUD base incentive for waves under 20 minutes
Ø 5 AUD additional for every extra 5 minutes
Use for primary data collection and calibration
Ø Rich profile for all new recruits including: range of primary and secondary
demographic variables, risk behaviours, health outcomes, Big5
personality inventory and social desirability indicators
7
9. www.srcentre.com.au
Panel establishment
• Single-frame design weights where
probability of selection is
•𝑝 =
!!!""
#!!$%!!
+
!"#&'
#"#
• where 𝑆"" is number of landline
respondents, 𝐿𝐿 is 0,1 indicator of having
landline, 𝑈"" is size of landline universe,
𝐴𝐷"" is number of adults in HH, 𝑆&' is
number of mobile respondents, 𝑀𝑃 is 0,1
indicator of having mobile and 𝑈&' is size of
mobile universe
2016recruitment
• Dual-frame RDD
• Landline (n=911)
• Mobile (n=2,411)
2016designweights
9
10. www.srcentre.com.au
Evolution of Life in Australia™ recruitment 2018 refresh
2016recruitment
• Dual-frame RDD
• Landline (n=911)
• Mobile (n=2,411)
• Panel size n=3,322
2018
replenishment
• Mobile RDD (n=267)
• Only age 54 and
under
• Quotas on university
education
• Panel size n=2,839
10
11. www.srcentre.com.au
Evolution of Life in Australia™ recruitment - 2019 refresh
2016recruitment
• Dual-frame RDD
• Landline
(n=911)
• Mobile
(n=2,411)
• Panel size
n=3,322 2018replenishment
• Mobile RDD
(n=267)
• Only age 54 and
under
• Quotas on
university
education
• Panel size
n=2,839
2019expansion
• ABS (n=1,810)
• Area-level
stratification by
SES and age
• Expand to 3,000
completes/ wave
• Panel size
n=4,025
11
12. www.srcentre.com.au
Evolution of Life in Australia™ recruitment
2016recruitment
• Dual-frame RDD
• Landline
(n=911)
• Mobile
(n=2,411)
• Panel size
n=3,322
2018replenishment
• Mobile RDD
(n=267)
• Only age 54 and
under
• Quotas on
university
education
• Panel size
n=2,839
2019expansion
• ABS (n=1,810)
• Area-level
stratification by
SES and age
• Expand to 3,000
completes/ wave
• Panel size
n=4,025
12
non-random attrition, increasingly unknowable probabilities of selection
decreasing weighting efficiency, increasing weighting complexity
14. www.srcentre.com.au
Derive exact probabilities of selection
Ø Potential to derive exact probabilities of selection either
o Assuming single frame or
o Treating as composite frames
Ø Challenges
o Future-proofing, increasingly complicated
o Large weight differentials leading to inefficiencies
o Attempted for 2018 – very low efficiency
o Not feasible for 2019 – too many non random elements
Ø References
o Kish, Leslie. 1965. Survey Sampling. New York: Wiley
o Lohr, Sharon L. 2011. ‘Alternative Survey Sample Designs: Sampling with Multiple
Overlapping Frames.’ Survey Methodology 37(2):197–213.
14
15. www.srcentre.com.au
Approximate probabilities of selection
Ø Propensity response model to match new cases (full coverage and
design weight applied) and continuing cases to approximate selection
weight
Ø Challenges
o Disproportionate selection of new cases (reference sample)
o Lack of common support for parts of the population, large weight differentials leading to
inefficiency
Ø References
o Rosenbaum, P.R. and D.B. Rubin (1983) The central role of the propensity score in
observational studies for causal effects. Biometrika 70(1), 41-55.
o Valliant, R., J. Dever, and F. Kreuter (2013). Practical Tools for Designing and Weighting
Survey Samples. New York: Springer.
o Bai, H., & Clark, M. (2019). Propensity score methods and Applications. Thousand Oaks,
CA: SAGE Publications, Inc.
15
16. www.srcentre.com.au
Propensity response model
16
A range of primary and secondary demographic variables, risk behaviours,
health outcomes, Big5 personality inventory and social desirability indicators
Propensity classes using stepwise logistic regression
Bias is measured as an absolute average error (difference) from the high
quality (official statistics) benchmarks for the modal response.
• Unweighted bias = 4.2
With propensity selection weights:
• Bias = 3.6
• Weighting Efficiency = 68.9%
17. www.srcentre.com.au
Doubly-robust approach
Ø Combine
o Approximate probabilities of selection
o Post-stratification (GREG)
Ø Challenges
o Calculation of probabilities of selection
o Model choice
o Further reduction in efficiency
Ø References
o Valliant, R. (2020) Comparing Alternatives for Estimation from Nonprobability Samples.
Journal of Survey Statistics and Methodology (2020) 8, 231–263
17
18. www.srcentre.com.au
Doubly-robust approach
Many combinations of variables (2,000 solutions for illustration):
Max weighting efficiency 69.1% for Bias < Unweighted Bias (4.1)
Minimum bias* 2.3 (Weighting efficiency 37.0%)
Ave weighting efficiency of 55.2% for solutions with bias btw 3. 5 & 3.7
*Variables used in calibration are excluded from bias measurement
18
19. www.srcentre.com.au
Super-population model
Ø Observed values are treated as if they have been generated by a model
o Combine all samples and
o Predict values for those units of the population that have not been sampled through the
whole panel using auxiliary information from external sources
Ø Challenges
o Model failure
o Tension between including covariates and efficiency
Ø References
o Valliant, R. and Dever, J. A. (2018) Survey Weights: A Step-by-Step Guide to Calculation.
Stata Press, 2018
19
20. www.srcentre.com.au
Comparison results – 2019 refresh panel weights
Approach Weighting
efficiency
Bias Measure
(unweighted
4.2)
Final variables used for
selection weight
Final variables used for
calibration
Approximate
selection weights
68.9% 3.6 Profile variables None
Doubly-robust
approach
Max 69.1%
37.0%
Ave 55.2%
4.1
Min 2.3
Btw 3.5 and 3.7
As above >2000 solutions
Super-population
model
Max 97.4%
53.9%
Ave 77.0%
Goldilocks
Solution:
75.8
4.1
Min 2.6
Btw 3.5 and 3.7
Goldilocks
Solution:
2.9
None >2000 solutions
Goldilocks Solution:
Age, Country of Birth,
Gender, Phone status,
SEIFA, State
20
21. www.srcentre.com.au
Lessons learnt
• Design-based weights are not suited for a long-term panel with multiple
refreshes
• Model-based approach is
• Simpler to implement
• Simpler to explain
• More efficient and less biased
• Universally applicable independent of the sample selection
mechanism
21
22. * PO Box 13328
Law Courts Victoria 8010
Thank you
( 03 9236 8500
A subsidiary of:
dina.neiger@srcentre.com.au
22