Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Ben Phillips, Senior Research Director, Survey Methodology (Social Research Centre)

1. A subsidiary of:
Alternatives to CATI (2):
Probability-based online panels
Telephone Interviewing in the Post-Modern Era Workshop
Melbourne
10 October 2019

2. www.srcentre.com.au
Authors
Dr Benjamin Phillips
Senior Research Director, Survey Methodology, Social Research Centre
2

3. www.srcentre.com.au
Rationale for probability-based
online panels
3

4. www.srcentre.com.au
Rationale for probability-based online panels
Best of both worlds: probability sample with reduced costs
Recruited offline using traditional sampling frames with good representation
of the target population (RDD, address-based sampling, area-probability
sampling, population registries)
Reduce costs by recycling sample (recruit once, use many times), reusing
demographics (fewer questions to ask) and reduce interviewer/paper costs
Larger panels provide access to sub-populations without screening costs
 Not yet at this scale in Australia
4

5. www.srcentre.com.au
Probability-based panels worldwide
Market Panels
U.S.
Gallup Panel; Ipsos KnowledgePanel; Pew Research Center American Trends Panel;
NORC AmeriSpeak; RAND American Life Panel; SSRS Opinion Panel; Washington
University of St. Louis The American Panel Survey; USC Understanding American Society
Other
countries
Australia (Life in Australia™); Canada (Probit); France (ELIPSS); Germany (GESIS Panel;
German Internet Panel); Iceland (Social Science Research Institute); Iran (IranPoll); Korea
(Korean Academic Multimode Open Survey); the Netherlands (LISS); New Zealand (in
development); Norway (Norwegian Citizen Panel); Sweden (Demoskop Panel; Novus
Sverigepanel; Sifo Panel); U.K. (NatCen Panel); pan-European (European Social Survey
CRONOS Panel)
Niche
Older adults: AARP (formerly American Association of Retired Persons) panel (U.S.);
Singapore Life Panel
Teachers and principals: RAND American Educator Panels
5

6. www.srcentre.com.au

7. www.srcentre.com.au
Recruiting Life in Australia™
7

8. www.srcentre.com.au
More information about recruiting Life in Australia™
Available at:
https://www.srcentre.com.au/our-
research/csrm-src-methods-papers
8

9. www.srcentre.com.au
Recruitment: Life in Australia™
Recruited October-December 2016 using dual-frame RDD (pilot: 60%
mobile, 40% landline; main: 70% mobile, 30% landline)
Replenished June-July 2018 using mobile RDD only with age restrictions (<
age 55), education quota (ceiling on university degree) and online only
 Address over-representation of older adults and those with university
degree
Replenishing September-December 2019 using mobile RDD and address-
based sampling
 Test functioning of address-based sampling
9

10. www.srcentre.com.au
Panel functioning
10

11. www.srcentre.com.au
Offline population: Life in Australia™
Includes offline population via CATI interviewing
On the agenda
 Encourage panel members to call in
 Test interactive voice response to reduce CATI costs
11

12. www.srcentre.com.au
Results of Life in Australia™
surveys
12

13. www.srcentre.com.au
Results: Life in Australia™
13
Rate What is it Rate
Recruitment rate
(RECR)
The rate at which people invited to join the panel initially
agree to participate
19.4%
Profile rate
(PROR)
The rate at which people who agreed to participate
completed the panellist profile and thus joined the panel
77.1%
Retention rate
(RETR)
The proportion of the original panellists who remain on
the panel at a specific wave of data collection
75.5%
Completion rate
(COMR)
The proportion of panellists invited to participate in a
specific wave who complete that wave’s questionnaire
79.1%
Cumulative response
rate 2 (CUMRR2)
RECR × PROR × RETR × COMR 8.9%
Cumulative response rate for Life in Australia™ is lower than European panels with in-
person recruitment, similar to U.S. panels with in-person recruitment and ahead of U.S.
panels with other types of recruitment

14. www.srcentre.com.au
Size of panel: Life in Australia™
Currently, Life in Australia™ has 2,590 active panellists
If we invited all panellists, we could complete approximately 2,000 interviews
 Limited ability to target sub-populations of interest at this size
Effective sample sizes accounting for design effects:
 Design effect = 2.60
 Effective sample size for n = 2,000: 770
14

15. www.srcentre.com.au
Characteristics: Life in Australia™ current panellists
Characteristic Benchmark Actual
Male 49.2% 46.5%
Female 50.9% 53.2%
Other 0.0% 0.3%
18-24 12.2% 6.9%
25-34 19.3% 14.1%
35-44 17.1% 15.3%
45-54 16.7% 18.0%
55-64 14.9% 18.7%
65-74 11.3% 18.5%
75+ 8.6% 8.2%
Characteristic Benchmark Actual
Less than uni 74.4% 56.9%
Uni degree 25.6% 43.1%
NSW 32.1% 29.7%
VIC 25.9% 25.6%
QLD 19.8% 19.3%
SA 7.1% 8.5%
WA 10.4% 11.0%
TAS 2.1% 2.4%
NT 1.0% 0.9%
ACT 1.7% 2.6%
15

16. www.srcentre.com.au
Panel conditioning: Life in Australia™
Panel conditioning is the impact of membership in a panel on attitudes and
behaviours
Compared responses of new panellists on the first post-replenishment wave
of Life in Australia™ (Monash/Scanlon Social Cohesion Survey) to veteran
panellists with adjustments for demographic differences between panellists
(age, education, gender, geography, volunteering and internet status) using
weights from coarsened exact matching and regression
Used Hochberg adjustment for multiple testing
No effect found on substantive responses or don’t know/refused responses
16

17. www.srcentre.com.au
Mode effects and transitions
17

18. www.srcentre.com.au
Mode effects and transitions
Move to web from phone likely to lead to mode effects
 Phone: response recency (more likely to select options read last), at
greater risk of response acquiescence and social desirability
 Paper/web: response primacy (more likely to select options appearing
earlier), straight-lining and other response non-differentiation
Prior to moving, rotate phone options to identify items affected by response
order effects (if you don’t already)
Strongly recommend bridging wave or waves fielded simultaneously across
both modes to determine effect of measurement on individual items
18

19. www.srcentre.com.au
Future developments
19

20. www.srcentre.com.au
Possible future developments
Context: minimum frequency of surveying to keep panel going
State-specific panels
Proprietary panels
 Pew Research Center American Trends Panel, German Internet Panel,
AARP panel
Specialist panels
 Teachers, health professionals, smokers, quitters
Longitudinal panel studies
20

21.  PO Box 13328
Law Courts Victoria 8010
 03 9236 8500
A subsidiary of:
21
Thank you
benjamin.phillips@srcentre.com.au