Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Darren Pennay - Founder and Executive Director, Research, Methods & Strategy and Ben Phillips - Senior Research Director, Survey Methodology (Social Research Centre)

1. A subsidiary of:
Improving Telephone Surveys – where we
should/n’t focus our efforts?
Telephone Surveying in the Post-Modern Era
Melbourne
10 October 2019

2. www.srcentre.com.au
Authors
Darren Pennay
Founder and Executive Director, Research, Methods & Strategy, Social Research Centre
Dr Benjamin Phillips
Senior Research Director, Survey Methodology, Social Research Centre
Karen Kellard
Executive Director, Qualitative Research Unit, Social Research Centre
Grant Lester
Director of Operations, Social Research Centre
2

3. www.srcentre.com.au
Pre-survey communications

4. www.srcentre.com.au
Developing the recruitment strategy for Life in AustraliaTM
4
CATI Intro Script
We tested: SMS, Primary Approach Letter (PAL), Logo, strapline, envelope
& CATI introductory script
SMS
Primary approach letter (PAL)
First reactions | Credibility | Impact on behaviour
Content, layout, credibility
Comprehension of the content | Likely response

5. www.srcentre.com.au
Method
5
 9 IDIs and 1 focus group
o Used concurrent and retrospective probing
o Used ‘read aloud’ and ‘interviewer
administered’ approaches
o Audio recorded, with note-takers
Data collection
 Grid approach to qualitative data
 Verbal feedback and brief report
Analysis & reporting

6. www.srcentre.com.au
Testing the SMS
6

7. www.srcentre.com.au
Testing the advance SMS
The text would arrive ‘out of the blue’ to people’s mobiles (with a local area
prefix).
Tested two options
You’ve been chosen to be
part of the Life in Australia
study http://src.is/LinA. An
interviewer will call to
provide more info. To
unsubscribe call
1800023040.
You’ve been selected to
be part of Life in Australia:
http://src.is/LinA. A
researcher will call to
provide further info. To
unsubscribe call
1800023040.

8. www.srcentre.com.au
Testing the SMS – main findings
 Probably would delete the text
 Wouldn’t click on the link as
wouldn’t trust it (spam)
 Prefer ‘researcher’ or ‘interviewer’?
Researcher
 Prefer ‘chosen’ or ‘selected’?
Selected
 Unsubscribe?
 No, as didn’t subscribe in the first place!
 Would it make any difference to
whether you later took our
call/took part in the research? Yes,
maybe
 Would like to see the word ‘study’
in the text
 Would like to see the caller
number or name
 Should mention our association
with the Australian National
University (ANU) for credibility
8

9. www.srcentre.com.au
Final SMS text
You’ve been chosen to be part
of the Life in Australia study
http://src.is/LinA. An interviewer
will call to provide more info. To
unsubscribe call 1800023040.The Social Research
Centre at Australian
National University
has selected you for
the Life in Australia
study. We will call
soon. To opt out call
1800023040.

You’ve been selected to be part
of Life in Australia:
http://src.is/LinA. A researcher
will call to provide further info.
To unsubscribe call
1800023040.

10. www.srcentre.com.au
Impact on response rates
AAPOR Response rates by advance text
message status
10
Text No text Comments
Response rate 3 33.3 30.2 Sig at 90% level
Cooperation rate 1 70.7* 60.3
Significantly higher cooperation rate for the advance text group based
on numbers to which calls were made. This makes sense as some of
the refusals were dealt with before the sample was loaded into CATI
Refusal rate 2 28.9* 23.3
Significantly higher refusal rate for the advance text message group
given the refusals received in advance of call attempts commencing.
Contact rate 3 65.2* 57.8
The contact rate is significantly higher for the advance text group due
the text eliciting some responses and to the ‘priming effect’ of the text
message.
Screening rate 76.6 89.4*
Lower screening rate for the advance text message sample due to
those who opt out of the survey in response to the advance text
message without every being screened for eligibility.
2014 VPHS
*z-test of column proportions: p≤.05.
Reply ‘1’ if you are a Victorian resident. Reply ‘2’ if
you are NOT a Victorian resident or Reply ‘3’ to Opt
Out. If you would like to contact the Department about
the survey please call 1800 882 028.

11. www.srcentre.com.au
Impact on fieldwork efficiency
Selected indicators of the fieldwork effort required to obtain an interview for
the advance text message and no advance text message group
11
Int.
Int.
hours
Int./Hr
Int. length
(mins)
Lapsed time
(mins.)
Time per int.
(mins)
Calls per
int.
Est. $AUD
per
interview
Advance
text
460 244 1.9 13.0 18.5 31.4 41.2 $24.81
No
advance
text
364 249 1.5 13.0 27.8 40.7 55.9 $29.09
17.3% reduction in fieldwork cost per interview
2014 VPHS

12. www.srcentre.com.au
Mental Health Experiences Survey
Response rates by advance text message status
12
National survey
*z-test of column proportions: p≤.05.
Advance text No advance text
A B
Response rate 3 15.9 17.2
Cooperation rate 1 52.0* 39.3
Refusal rate 2 37.9* 27.9
Contact rate 3 56.1* 47.9
Screening rate 50.2* 71.5
Reply ‘1’ to Opt Out. If you would
like to contact the Social Research
Centre about this study please call
1800 023 040.”

13. www.srcentre.com.au
Mental Health Experiences Survey
Selected indicators of the fieldwork effort required to obtain an interview for
the advance text message and no advance text message group (MHES)
13
*The longer interview is due to topic salience. The text mentioned that the survey was about mental health discrimination. This means that, relative to
the no pre text sample, more people with a mental health issue opted in. This group had a slightly longer pathway through the questionnaire.
Int.
Int.
hours
Int./Hr.
Int.
length
(mins.)
Lapsed
time
(mins.)
Time per
int.
(mins.)
Calls
per int.
Est. $
per
interview
Advance
text
1329 729.2 1.8 20.0* 12.6 32.9 22.4 $22.11
No
advance
text
1302 762.0 1.7 18.9 16.2 35.1 28.0 $24.00
8.5 % reduction in fieldwork cost per interview

14. www.srcentre.com.au
Summary of findings / implications from these two studies
14
 Sending advance text messages
o Improves fieldwork efficiency (even when no benefit from geographic screening)
o Increases refusal rates
o Increases contact rates
o Increases cooperation rates
o Has a negligible impact on overall response rates
 Nearly half (48%) of respondents to the MHES who recalled receiving an
advance text message said that they would not have otherwise
participated in the survey
 Next steps include:
o Further research into the wording of text messages
o Further research into non-response bias arising from the ‘screening’ and ‘opt out’ processes

15. www.srcentre.com.au
Testing the Primary Approach Letter (PAL)
15

16. www.srcentre.com.au
Testing the Intro Script
19

17. www.srcentre.com.au
Professional PR & comms
21

18. www.srcentre.com.au
Pre-survey letter
To this
22
From this

19. www.srcentre.com.au
Reminders
To this
23
From this

20. www.srcentre.com.au
Questionnaire booklet
To this
24
From this

21. www.srcentre.com.au
Where is this leading – possible future developments
 More research into PALs and pre-survey SMS
 Move from passive to active pre survey communications
o Conditional incentives for push to web
o Click here to complete the survey / Go online to make an appointment
o Promote inbound calls. Dial during business hours and me connected directly to an
interviewer
 Develop surveys like ‘brands’ but not too funky!
25

22. www.srcentre.com.au
Predictive diallers
26

23. www.srcentre.com.au
Predictive dialling
27
Selected statistics Preview /auto-dial Predictive mode % change Cost impact
Calls per hour 42.5 54.3 28%
Refusals per hour 1.9 2.8 47%
Interviews per hour 2.6 2.1 -19%
Implications
 ‘Dead air’ has an impact on predisposition to cooperate
 Ask your supplier what dialler settings they are using and what proportion of nuisance calls
they are making on your behalf!
 Is answering machine detection enabled?
 Can ‘dead air’ be monitored and reported on
 SRC is hoping to do an experimental survey and will circulate the results to this group and
others
 Time series implications for call dispositions – seem to get a lot of unknown / unexplained call
outcomes
 False positives & false negatives
Caveat **Results from pre-production testing**

24. www.srcentre.com.au
Extended call routines
Dr Benjamin Phillips
Senior Research Director, Survey Methodology
28

25. www.srcentre.com.au
Case study
Characteristic Landline Mobile Total
Interviews 16,941 16,666 33,607
Sample used 214,718 94,056 308,774
Dials made 717,923 409,375 1,127,298
29

26. www.srcentre.com.au
Response rate if stopped at call attempt (2017)
30
0%
5%
10%
15%
20%
25%
1 2 3 4 5 6 7 8 9 10 11 12 13
Responserate(AAPORRR3)
Call attempt
Mobile
Landline

27. www.srcentre.com.au
Minutes per completed interview by frame and attempt
31
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13
Minutespercompletedinterview
Call attempt
Mobile
Landline

28. www.srcentre.com.au
Landline number of attempts to stabilise frequencies
Attempts Variables
Always
stable
Region, smoker, vegetable consumption, fruit consumption, depression (K10), general
health status, life satisfaction, diagnosed with anxiety or depression, dental health,
general trust
1
Country of origin, education, obesity, feel life is worthwhile, diagnosed with diabetes,
diagnosed with stroke
2 Sex, sedentary (weekdays), support for multiculturalism
3 Language spoken at home, not visited dentist due to cost
4
Sedentary (weekends), diagnosed with heart disease, diagnosed with cancer, diagnosed
with osteoporosis, diagnosed with arthritis
5
Alcohol (lifetime risk), alcohol (risk at drinking incident), diagnosed with hypertension,
diagnosed with multiple chronic conditions
6 Household income
7 Age
8 Employment, children under age 18 in household
32

29. www.srcentre.com.au
Mobile number of attempts to stabilise frequencies
Attempts Variables
Always
stable
Sex, smoker, vegetable consumption, fruit consumption, obesity, sedentary
(weekdays), sedentary (weekends), alcohol (lifetime risk), depression (K10), general
health, life satisfaction, life is worthwhile, diagnosed with diabetes, diagnosed with
heart disease, diagnosed with stroke, diagnosed with osteoporosis, dental health, not
visited dentist due to cost, general trust, support for multiculturalism
1 Diagnosed with anxiety or depression
2 Region, education, alcohol (risk at drinking incident), diagnosed with cancer
3
Country of birth, diagnosed with hypertension, diagnosed with arthritis, diagnosed with
multiple chronic conditions
4 Age, language, household income
5 Employment, children under age 18 in household
33

30. www.srcentre.com.au
Children under 18 in household
34
0%
10%
20%
30%
40%
1 2 3 4 5 6 7 8 9 10 11 12 13
Percentagewithchildunder18
Call attempt
Mobile
Landline

31. www.srcentre.com.au
Employment status
35
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
Employed Unemployed Not in labour force
Mobile Landline
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
Employed Unemployed Not in labour force

32. www.srcentre.com.au
Age
36
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
18-24 25-34 35-44 45-54 55-64 65-74 75+
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
18-24 25-34 35-44 45-54 55-64 65-74 75+
Mobile Landline

33. www.srcentre.com.au
Household income
37
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
Less than $40,000 $40,000 to < $100,000 $100,000+
Mobile Landline
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
Less than $40,000 $40,000 to < $100,000 $100,000+

34. www.srcentre.com.au
Hypertension
38
0%
10%
20%
30%
40%
50%
60%
1 2 3 4 5 6 7 8 9 10 11 12 13
Percentagewithchildunder18
Call attempt
Mobile
Landline

35. www.srcentre.com.au
Multiple conditions
39
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
No chronic disease 1 chronic disease 2+ chronic diseases
Mobile Landline
0%
20%
40%
60%
80%
100%
1 2 3 4 5 6 7 8 9 10 11 12 13
Call attempt
No chronic disease 1 chronic disease 2+ chronic diseases

36. www.srcentre.com.au
Conclusions
Analyse your data to see if your important estimates change
Potential for adaptive design: terminate study when estimates stabilise
(‘phase capacity’)
Savings from call cycle reduction can be put into response maximisation or
bias minimisation
 Pre-survey communications
 Incentives
40

37. www.srcentre.com.au
Refusal conversion
Dr Benjamin Phillips
Senior Research Director, Survey Methodology
41

38. www.srcentre.com.au
Descriptive statistics
33,654 completes
1,321 (3.9%) from refusal conversion
42

39. www.srcentre.com.au
Age
4% 4%
9% 5%
11%
9%
16%
15%
21%
19%
23%
23%
16%
24%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
75+
65-74
55-64
45-54
35-44
25-34
18-24
43

40. www.srcentre.com.au
Employment status
51%
45%
3%
3%
46%
53%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
Not in labour force
Unemployed
Employed
44

41. www.srcentre.com.au
Household income
35%
41%
38%
36%
28% 23%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
$100,000+
$40,000 to $99,999
Less than $40,000
45

42. www.srcentre.com.au
Education
2% 2%
35%
43%
63%
54%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
Tertiary
Secondary
Primary
46

43. www.srcentre.com.au
Alcohol: lifetime risk
21%
27%
20%
20%
59%
53%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
Increased risk
Reduced risk
Abstainer/no longer drinks
47

44. www.srcentre.com.au
Alcohol: short-term risk
21%
27%
41%
41%
38%
32%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
Increased risk
Reduced risk
Abstainer/no longer drinks
48

45. www.srcentre.com.au
Avoided or delayed visiting a dental professional due to cost
29% 25%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
Yes
49

46. www.srcentre.com.au
Multiculturalism makes life in your area better
8% 10%
3%
3%
28%
27%
50% 43%
11% 16%
0%
20%
40%
60%
80%
100%
Not refusal conversion Refusal conversion
Not applicable
Yes, definitely
Sometimes
Not often
Not, not at all
50

47. www.srcentre.com.au
Summary
Not an airtight case for the benefits of refusal conversion
 Older (survey respondents are already elder-biased)
 Less likely to be in labour force
 Lower income
Does improve educational profile, but likely linked to age
Carefully consider amount of effort devoted to refusal conversion
Where non-response related to survey topic, can use brief questions to non-
respondents to adjust estimates: discussion in later session
51

48.  PO Box 13328
Law Courts Victoria 8010
 03 9236 8500
A subsidiary of:
52
Thank you
darren.pennay@srcentre.com.au
benjamin.phillips@srcentre.com.au