Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy (Social Research Centre)

1. A subsidiary of:
Alternatives to CATI –
Non-probability online panels
Telephone Interviewing in the Post-Modern Era Conference
Melbourne
10 October, 2019

2. www.srcentre.com.au
Background
Conditions in Australia are good for online research …
In 2016-17, 86% of Australian households had access to the internet at home
2
Source: ESOMAR
 Online research is
continuing to grow in
popularity
 50+ commercial ‘online
research panels’
 First of these established in
the late 1990’s
 Non-probability sampling
methods are ubiquitous
Percent of research industry turnover by mode of data
collection: CATI and online

3. www.srcentre.com.au
Cons
 Self-selection bias
 Non-coverage – no sampling frame
 Reliance on the computer-literate
respondents
 Weighting can increase bias!
 Standard errors / confidence
intervals mis-represented
 Generalising to the target
population is problematic
 Professional respondents, fake
accounts and Bots
Pros
 Reduced cost
 Improved timeliness
 Respondent convenience
 Reduced social desirability bias
 Can target ‘hard to reach’
populations
 Multimedia functionality
 Computerised questionnaire
scripts
3
Pros and cons of non-probability panels

4. www.srcentre.com.au
Probability and nonprobability respondents
4
Source: Fahimi et al., 2015
Early adopter
•I usually try new products before
other people
•I often try new brands
•When I shop I look for what is
new
•I like to be among the first to try
something new
•I like to tell others about new
brands or technology
Social Engagement
• Take vacation with others
• Exercise with others
• Have meals with others
Self-Importance
• Sharing opinions
• Importance of opinions
• Confidence
Shopping Habits
• Use coupons for shopping
• Enjoying shopping online
• Brand compared to price
Happiness and
Security
• Happiness with life
• Feeling insecure and
lonely
• Cyber security concerns
Political Views
• Influence on national
politics
• Government effectiveness
• Following the news
Sense of Community
• Feel part of the community
• Relocations in recent years
• Religiosity
Altruism
• Donating blood
• Donating items
• Volunteering without pay
Survey Participation
• Experience with surveys
• Important of opinion
surveys
• Frequency of online
surveys
Internet &Social Media
• Frequency of personal
email
• Frequency of web search
• Time spent watching TV

5. www.srcentre.com.au
ARF Study, 2009 – Intention to buy new soup
5

6. www.srcentre.com.au
ARF 2013, Have you ever watched a video on YouTube
6
Source: Weighted results from 17 online panels and 5 online river samples from the ARF 2013 online panel comparison experiment. Sample size 938-1,376. Mean 1,185

7. www.srcentre.com.au
The Online Panels Benchmarking Study
 Three surveys based on probability samples of the Australian population aged 18 years
and over and five surveys of persons aged 18 years and over administered to members
of non-probability online panels
 Survey questionnaire included a range of demographic questions and questions about
health, wellbeing and use of technology
 Same questions were used across the eight surveys (Unified approach to questionnaire
design to try and minimise mode effects)
 9 minutes average interview length for online and telephone
o 12 page booklet for the hard copy version
 Fieldwork Oct – Dec 2015
 Data and documentation available from the Australian Data Archive and the full report is
available as one of our CSRM-SRC Methods Papers
https://www.ada.edu.au/ada/01329
8

8. www.srcentre.com.au
Methods – questionnaire content
Items were chosen because high quality population benchmarks were available
Primary demographics (6 variables)
 Sex, age, location, educational attainment, country of birth and telephone status
Secondary demographics (13 variables)
 Indigenous status, socioeconomic status, citizenship, enrolled to vote
 Geographic mobility, home ownership status, household composition, geographic remoteness
 Language spoken at home, internet access, volunteerism, employment status, wage and salary income
Substantive measures (6 variables)
 General health status, psychological distress, life satisfaction
 Private health insurance coverage, daily smoking status and alcohol consumption in the last 12 months
Calibration and other variables
 Early adopter characteristics
 Use of information technology – internet use and online survey participation
10

9. www.srcentre.com.au
Methods – sampling frames
11
Probability Surveys
Survey 1 - Address Based Sampling (A-BS)
 Geocoded National Address File. 91% match to the Postal Address File
 36% match to landline. 14% mobile match
Survey 2 - Recruited at the conclusion of an established dual-frame RDD CATI survey
(ANU Poll)
 60:40 landline / mobile blend. n=1,200. (58% agreed to be re-interviewed)
 Depending upon preference, sample members were emailed a link to complete the
subsequent survey online or sent a hard copy questionnaire to return via the mail
Design 3 - Standalone dual-frame RDD CATI Survey (DFRDD)
 50:50 landline / mobile blend.

10. www.srcentre.com.au
Methods
12
Non-probability Online Panels
 RFQ sent to eight companies (six replied / five selected)
 Asked to conduct a “nationally representative” survey of 600 respondents from their
respective panels. No instructions provided
 Four of the panel providers addressed this task by moving the age, gender and location
questions to the beginning of the questionnaire and using these questions as screeners
 Remaining panel provider drew its sample to be ‘Australian Bureau of Statistics
representative’ and applied quotas allowing for +/- 5% variation
 Price was not a selection criterion. 14% range between cheapest and most expensive.

11. www.srcentre.com.au
Results – secondary demographics
Main error metric: Average absolute error: The average difference (percentage points)
across all benchmarks between the official statistics and the survey estimates. The modal
response category for each variable was used.
Error metrics across 13 secondary demographics
18
Probability Surveys Non-probability Panels
ABS
ANU
Poll
DFRDD P1 P2* P3 P4* P5*
Average absolute error 5.75 5.87 5.71 6.05 4.93 5.89 6.18 6.84
Largest absolute error 11.66 11.57 11.24 15.22 14.38 13.73 13.15 13.70
No. of significant
differences from
benchmarks (out of 13)
7 7 8 7 8 6 5 4
* ISO 26362 accredited Online Access Panel

12. www.srcentre.com.au
Results – substantive health characteristics
19
Substantive variables
Benchmark
value
Distance from benchmarks
Probability Non-probability
ABS
ANU
Poll
DF
RDD
P1 P2* P3 P4* P5*
Life satisfaction (8 out of 10)
Percentage point error 32.6 -2.0 -2.0 1.9 -11.9 -11.6 -4.5 -9.2 -7.9
Psychological distress - Kessler 6 (Low)
Percentage point error 82.2 -10.6 -11.6 -8.1 -25.9 -23.5 -22.2 -25.0 -23.2
General Health Status (SF1) (Very good)
Percentage point error 36.2 0.4 -2.0 -2.6 -4.1 -5.8 -5.3 -5.0 1.5
Private Health Insurance
Percentage point error 57.1 3.4 1.9 3.3 -8.9 -12.5 -3.7 -0.6 -2.6
Daily smoker
Percentage point error 13.5 -4.1 3.5 1.6 9.8 6.7 3.9 2.7 4.3
Consumed alcohol in the last 12 months
Percentage point error 81.9 3.6 -2.8 -4.0 2.4 5.3 3.9 4.2 1.5
* ISO 26362 accredited Online Access Panel

13. www.srcentre.com.au
Results – substantive health characteristics
Average error across six substantive measures
20
Probability Surveys Non-probability Panels
ABS
ANU
Poll
DFRDD
P1
$$
P2*
$
P3
$$$$$
P4*
$$$
P5*
$$$$
Avge. error 4.02 3.98 3.58 10.5 10.9 7.24 7.78 6.83
Largest absolute error 10.59 11.57 8.08 25.86 23.52 22.20 24.96 23.20
No. of significant
differences from
benchmarks (out of 6)
2 3 2 4 6 3 3 3
* ISO 26362 accredited Online Access Panel

14. www.srcentre.com.au
Results – substantive health characteristics
Impact of weighting on average absolute error
21
Probability Surveys Non-probability Panels
ABS
ANU
Poll
DFRDD P1 P2* P3 P4* P5*
Unweighted (avge. error) 4.28 3.68 4.63 9.34 10.35 7.28 7.20 6.41
Weighted (avge. error) 4.02 3.98 3.58 10.50 10.90 7.24 7.78 6.83
Impact      -  
* ISO 26362 accredited Online Access Panel

15. www.srcentre.com.au

16. www.srcentre.com.au
How does Life in AustraliaTM perform?
23
Probability surveys Non-probability panel surveys
ABS
ANU
Poll
RDD
Life in
Australia#
P1 P2 P3 P4 P5
Secondary demographics (12)* 5.74 5.77 5.88 5.41 5.48 4.31 5.38 5.59 6.27
Substantive variables (6) 4.02 3.98 3.58 4.41 10.50 10.90 7.24 7.78 6.83
Combined (18 variables) 5.17 5.17 5.11 5.08 7.15 6.51 6.00 6.32 6.46
Rank 3 3 2 1 9 8 5 6 7
# Notes on weighting Life in Australia sample
The approach to deriving weights consisted of the following steps:
1. Compute a base weight for each respondent as the product of two weights:
a) Their enrolment weight, accounting for the initial chances of selection and subsequent post-stratification to key
demographic benchmarks; and
b) Their response propensity weight, estimated from enrolment information available for both respondents and non-
respondents to the Online Panels Benchmarking Survey;
2. Calibrate the base weights so that they satisfy the latest population benchmarks for several demographic characteristics; and
3. Trim the weights so that no respondents have an undue influence on estimates made from the dataset.
*Analysis limited to 12 secondary demographics. ‘Internet access’ dropped as included as a
post stratification variable when weighting the Life in Australia panel

17. www.srcentre.com.au
Initial conclusions
 Probability surveys and the non-probability online panels perform similarly well with
respect to producing reasonably accurate measures of the population distribution in
terms of secondary demographics
 Despite this, in terms of accurately reflecting the population distribution on selected
health characteristics the probability surveys even with modest response rates are, on
average, more than twice as accurate (2.24 times) than non-probability panels
A representative demographic profile does not predict accuracy
Kennedy, Mercer, Keeter et al. (Pew, May, 2016)
 Across numerous studies, probability based samples are more accurate in comparison
to benchmarks than non-probability based panels (Callegaro et al. , 2014)
24

18. www.srcentre.com.au
Implications for researchers using non-probability panels
 Threshold question - Are non-probability online panels fit-for-purpose?
 Weighted or unweighted? Standard post stratification can lead to more biased estimates
 Adding uncommon variables to the weighting solution can help. But which ones and in
which circumstances?
 Include high quality benchmarks in your study to try and get some idea of bias
 Refrain from changing panels when conducting continuous tracking research (Vonk,
Ossenbruggen & Willems, 2006)
 “A web-based panel does not provide a representative sample, and secondly that
different panels produce different results” (Crasweller Roger & Williams, 2008)
 “The findings suggest strongly that panels are not interchangeable” (Walker, Pettit & Rubinson,
2009)
 Hybrid designs: Calibration, sample blending and sample matching seem like
promising techniques to reduce error in non-probability surveys in some circumstances
28

19. www.srcentre.com.au
Next presentations
Emerging alternatives
 Probability-based online panels
 Address-based sampling coupled with push to web methods
29

20. www.srcentre.com.au
Questions?
Telephone Interviewing in the Post-modern Era
Melbourne, 10 October 2019
30

21. www.srcentre.com.au
References
Australian Bureau of Statistics, (2018). Household use of information technology, Australia, 2016–17 [machine-readable data file], cat. no. 8146.0, ABS, Canberra.
Baffour, B., M. Haynes, M. Western, D. Pennay, S. Misson and A. Martinez (in press). Weighting strategies for combining data from dual-
frame telephone surveys: emerging evidence from Australia. Journal of Survey Statistics and Methodology.
Baker R, Brick JM, Bates NA, Battaglia M, Couper MP, Dever JA, Gile KJ, & Tourangeau R. (2013) Report of the AAPOR Task Force on Non-
Probability Sampling. American Association for Public Opinion Research
Chang, L. and Krosnick, J. (2009). National Surveys Via RDD Telephone Interviewing Versus the Internet: Comparing Sample
Representativeness and Response Quality. Public Opinion Quarterly, 73(4), pp.641-678.
ESOMAR Global Market Research, 2017
Fine, Brian, (2016, July). Online Research Panels Around the World: The Situation in Australia. Presented at the Current State and Future
of Online Research in Australia Workshop, Canberra, Australia. http://www.srcentre.com.au/online-research
Greenbook Research Industry Trends Report 2017(www.Greenbook.org.grit)
Kennedy, C., Mercer, A., Keeter, S., Hatley, N., McGeeney, K., and Gimenez, A. (2016). Evaluating online non-probability surveys. Pew
Research Center. Retrieved from http://www.pewresearch.org/2016/05/02/evaluating-online-non-probability-surveys/
Pennay, D., Neiger, D., Lavrakas, P. J., & Borg, K. (2018). The Online Panels Benchmarking Study: a total survey error comparison of
findings from probability-based surveys and non-probability online panel surveys in Australia. Australian National University.Walker, R., Pettit,
R. and Rubinson, J. (2009). The Foundations of Quality Initiative. Journal of Advertising Research, 49(4), pp.464-485.
Yang, Y., Callegaro, M., Villar, A., Chin, T., & Krosnick, J. A. (2016). Assessing the Accuracy of 51 nonprobability online panels and river
samples: A study of the Advertising Research Foundation 2013 online panel comparison experiment.
Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levendusky, M. S., Simpser, A., (2011). Comparing the Accuracy of RDD Telephone
Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples. Public Opinion Quarterly
31

22.  PO Box 13328
Law Courts Victoria 8010
Thank you
 03 9236 8500 PO Box 13328
Law Courts Victoria
Thank you
A subsidiary of:
Darren Pennay
darren.pennay@srcentre.com.au
32