ESRA 2015 postal survey is dead, long live the postal survey

1
Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI
Paste co-
brand logo
here
Paste co-
brand logo
here
Gerry Nicolaas & Patten Smith
ESRA conference, 14-17 July 2015, Reykjavik, Iceland
16/07/2015
The postal survey is dead,
long live the postal survey!
Part one: Spending more to save money.
Part two: Is more always better?

2
Paste co-
brand logo
hereCo-authors
Our colleagues at Ipsos MORI
• Kevin Pickering (Head of Statistics)
• Will Scott (Survey project manager)
• Lexi Kong (Survey researcher)
Our client at NHS England
• Chris Branson

3
Paste co-
brand logo
hereOverview
1. Background
2. Design of the experiment
3. Response Rates
4. Implications for Cost and Quality
5. Conclusions

4
Paste co-
brand logo
here
Background

5
Paste co-
brand logo
hereGeneral Practice Patient Survey (GPPS)
• A survey run by Ipsos MORI on behalf of NHS1 England
• Patients’ overall experience of primary care health services
• Results are used by NHS England to update indicators for key
national initiatives aimed at
• Accountability
• Performance monitoring
• Health outcome improvement
1. NHS – National Health Service

6
Paste co-
brand logo
hereDesign of GPPS
• Sampling frame = All patients registered with a GP in England
• Statistically robust results needed at practice level as well as national
• In 2014/15 the issued sample size was 2.16 million patients
• Two waves of data collection per year
• Data collection mode = postal questionnaire (8 pages)
• Large sample & timely results constrain the mail-out strategy:
1. Cover letter with questionnaire
2. One month later - reminder letter + replacement questionnaire
3. One month later - 2nd reminder letter + replacement qnaire
• Option to complete online (only 4% of completed questionnaires)

7
Paste co-
brand logo
hereGPPS 2006-2014 : Declining response rates
2 pp.
4 pp.
8 pp.

8
Paste co-
brand logo
hereTailored Design Method (TDM)
• Developed over 40 years ago by Professor Don Dillman and since then
regularly revised by himself and colleagues
• TDM guidelines for postal survey implementation
• Personalisation
• Multiple contacts; e.g. up to 5 contacts
• Vary each contact; e.g. pre-notice letter, postcard reminder after 1st
questionnaire mailing, tap into different motivations for participation,
different modes of contact
• Replacement questionnaires
• Postage paid return envelopes using real stamps
• Respondent incentives
• etc.
Reference: Dillman, D.A., Smyth, J.D. & Christian, L.M. (2014). Internet, Phone, Mail, and Mixed-
Mode Surveys: The Tailored Design Method.

9
Paste co-
brand logo
hereEvaluating the GPPS implementation using TDM
Already implemented on
GPPS
Not feasible for GPPS Could be implemented on
GPPS
Personalisation Real stamps (sample too large) Add a pre-notice letter
Multiple contacts (but only 3) Respondent incentives (public
perception)
Add a postcard reminder
Replacement questionnaires
(with all 3 mailings)
Redesign the letters so that
they tap into different
motivations
Various other TDM guidelines
such as assigning unique ID
numbers, procedures for dealing
with queries, etc.
Reduce respondent burden by
reducing the questionnaire
length

10
Paste co-
brand logo
hereOriginal cover letters for GPPS (1st, 2nd & 3rd mailing)
Very similar:
• Same motivator = appeal for help to improve health services;
• 2nd letter adds we want to hear from as many people as possible;
• 3rd letter adds that this is the last opportunity

11
Paste co-
brand logo
Very similar;

12
Paste co-
brand logo
Very similar;

13
Paste co-
brand logo
Very similar;

14
Paste co-
brand logo
here
Design of experiments

15
Paste co-
brand logo
here2 experiments
1. Mailout strategy
A. Add pre-notice letter, sent 1 week before 1st questionnaire mailing
B. Add postcard reminder, sent 1 week after 1st questionnaire mailing
C. Redesigned cover letters based on TDM & behavioural insights
2. Shortened questionnaire
D. 4 pages rather than the current 8 pages

16
Paste co-
brand logo
hereExperiment 1: mailout strategy
C. Co ve r le tte rs
A. Pre-notice B. Postcard Sta nda rd Re de sig ne d
No
No 1.3 million 3,000
Ye s 3,000 3,000
Ye s
No 3,000 3,000
Ye s 3,000 3,000

17
Paste co-
brand logo
hereExperiment 2: shorter questionnaire (4 pages)
• Fresh sample of patients run in parallel with main survey
• So as not to sacrifice range of data collected for the main survey
• Two experimental conditions:
1. Short questionnaire
2. Short questionnaire plus all 3 changes included in experiment 1
Mailout Strategy
Sta nda rd
Sta nda rd +
pre -no tic e , p o stc a rd &
re de sig ne d le tte rs
D. 4 pa g e
que stio nna ire
3,000 3,000

18
Paste co-
brand logo
hereAdded value to the field of survey methodology
• Sample size is very large:
• Multiple treatments can be tested using a factorial design
• Sample sizes for treatment groups are large (3,000 each)
• Sample is more or less equivalent to general population
• Assess the cost implications
• Assess non-response bias

19
Paste co-
brand logo
here
Design of prenotice letter,
postcard and cover letters.

20
Paste co-
brand logo
hereA. Pre-notice letter
• Positive & timely notice that the patient will be receiving a request
to help with an important study
• Guiding principles for design of pre-notice letter:
• NHS England logo & signed by the survey client;
• Brief, personal, positively worded;
• Aimed at engendering enthusiasm & building anticipation rather
than providing details or conditions for the survey;
• Sent within one week of the first questionnaire mail-out.

21
Paste co-
brand logo
hereA. Pre-notice letter

22
Paste co-
brand logo
hereB. Postcard reminder
• Sent to all sampled patients as a thank you for completing the
questionnaire or a nudge to do so
• Guiding principles for design of postcard reminder:
• NHS England logo & signed by the survey client;
• Aimed at jogging memories and rearranging priorities;
• Sent one week after the 1st questionnaire mail-out.

23
Paste co-
brand logo
hereB. Postcard Thank You/Reminder

24
Paste co-
brand logo
hereC. Cover letters
• Short and simple to read
• Communicate importance of survey request:
• NHSE logo
• Signature of NHSE client
• Professional
• Personalised
• Avoid visual clutter
• Motivational statements should vary across 3 letters:
1. NHS England needs your help;
2. Taking part will help you, your family and your community;
3. Last chance to help shape the health services in your area
(including end date).

25
Paste co-
brand logo
hereOriginal & redesigned cover letter for GPPS (1st mailing)

26
Paste co-
brand logo

27
Paste co-
brand logo

28
Paste co-
brand logo
hereRedesigned cover letters for GPPS (1st, 2nd & 3rd mailing)

29
Paste co-
brand logo

30
Paste co-
brand logo

31
Paste co-
brand logo

32
Paste co-
brand logo
here
Results:
Response rates

33
Paste co-
brand logo
hereResponse by experimental treatment & month
Re sp o nse ra te s
Mo nth 1 Mo nth 2 Mo nth 3
( fina l)
Co ntro l ( 3 q na ire m a iling s) 21.9% 29.1% 32.7%
A. Pre -no tic e le tte r 23.5% 29.5% 32.9%
B. Po stc a rd 28.0% 35.1% 37.8%
C. Re d e sig ne d le tte rs 20.4% 29.0% 34.4%
D. Sho rte r q ue stio nna ire 23.9% 31.8% 34.6%
A + B 29.5% 36.1% 38.6%
A + C 22.8% 30.2% 34.9%
B + C 28.0% 37.3% 41.5%
A + B + C 30.0% 36.9% 40.3%
A + B + C + D 34.2% 40.3% 43.6%
+5.1
+10.9

34
Paste co-
brand logo
hereMultiple Linear Regression (STATA)
• Pre-notice letter has no statistically significant effect on final response rate
• Postcard has the greatest effect on final response rate
• No interactions – the effects of a postcard, redesigned letters, and shortened
questionnaire are additive
Co e ffic ie nts
@
Co e ff. St. Erro r t Sig n.
Co nsta nt 0.327
A. Pre -no tic e le tte r 0.000 0.005 0.06 0.949
B. Po stc a rd 0.058 0.005 11.14 0.000
C. Re de sig ne d le tte rs 0.022 0.005 4.23 0.000
D. Sho rte r que stio nna ire 0.023 0.006 3.60 0.000
@Dependent variable: Final response rate
0.103
0.430

35
Paste co-
brand logo
here
Does the postcard increase response because it is
merely an extra contact attempt?
Month 1
Month 3
Month 2
Control Group Add Postcard (B)
Letter with
questionnaire
Letter with
questionnaire
Letter with
questionnaire
Letter with
questionnaire
Letter with
questionnaire
Postcard
1 1
2 3
2
3
Resp rate = 32.7%
Letter with
questionnaire
4
Resp rate = 37.8%

36
Paste co-
brand logo
here
Does the postcard increase response because it is
merely an extra contact attempt?
Month 1
Month 3
Month 2
Control Group Add Postcard (B)
Letter with
questionnaire
Letter with
questionnaire
Letter with
questionnaire
Letter with
questionnaire
Letter with
questionnaire
Postcard
1 1
2 3
2
3
Resp rate = 35.1%
Resp rate = 32.7%

37
Paste co-
brand logo
hereSummary of results
Pre-notice letter Postcard reminder Redesigned
cover letters
Shortened
questionnaire
• No impact on final
response rate
• Positive impact
on final response
• 5.8 percentage
points
• Positive impact
on final response
• 2.2 percentage
points
• Positive impact
on final response
• 2.3 percentage
points
• No interactions – in other words, the effects of the postcard, the redesigned letters and
the shortened questionnaire on the final response rate are additive.
• Postcard + redesigned letters + shortened qnaire = 10.3 percentage points

38
Paste co-
brand logo
here
Implications for cost and quality

39
Paste co-
brand logo
hereGetting value for money
• For a fixed cost we wish to minimise survey error:
• Maximise precision
• Minimise bias
• We also prefer to collect more rather than less data
• In practical terms, we want to spend our money in a way which:
1. Increases our sample size
2. Increases our response rate (assuming this will decrease NR
bias)
3. Allows us to maintain current questionnaire length

40
Paste co-
brand logo
hereGetting value for money
• Following analyses do two things:
1.Test which experimental treatment best meets these
three desiderata
2.Test validity of the embedded assumption that increasing
response rate will decrease NR bias

41
Paste co-
brand logo
here
Costs

42
Paste co-
brand logo
hereCalculation of indicative costs
• Costs = printing, despatch, postage and scanning
• Unit cost estimates for:
• Mailing out an 8-page questionnaire
• Mailing out a 4-page questionnaire
• Mailing out a pre-notice letter
• Mailing out a postcard reminder
• Return mailing and scanning of completed questionnaire
• Calculate an indicative cost per productive for each experimental condition group.
• Explore relationships between cost, response rate and achieved sample size:
• If the response rate is higher in months 1 & 2, then fewer reminders will be sent out and more
completed questionnaires will be returned.
• If the final response rate is higher, then a smaller sample size can be issued to achieve the same
number of completed questionnaires.
• Pre-notice letters & postcards incur extra cost that needs to be offset by an increase in response
coupled with a reduction in the issued sample size & number of reminders in order to be cost
neutral.
• The redesigned letters and shorter questionnaire do not incur extra costs.

43
Paste co-
brand logo
hereWhat we get for a fixed budget
Cost per achieved
(control indexed at
100)
Achieved
sample
size(control
indexed at
1,000) Response rate
Control group 100.00 1,000 32.7%
A - prenotificaton letter 126.76 789 32.9%
B - Postcard reminder 106.35 940 37.8%
C - Redesigned letters 96.37 1,038 34.4%
D - Short questionnaire 91.89 1,088 34.6%
AB 127.23 786 38.6%
AC 120.20 832 34.9%
BC 97.43 1,026 41.5%
ABC 122.21 818 40.3%
ABCD 110.61 904 43.6%

44
Paste co-
brand logo
hereWhat you get for a fixed budget – ordered by cost
Cost per achieved
(control indexed at
100)
Achieved
sample
size(control
indexed at
1,000) Response rate
D - Short questionnaire 92 1,088 34.6%
C - Redesigned letters 96 1,038 34.4%
BC 97 1,026 41.5%
Control group 100 1,000 32.7%
B - Postcard reminder 106 940 37.8%
ABCD 111 904 43.6%
AC 120 832 34.9%
ABC 122 818 40.3%
A - prenotificaton letter 127 789 32.9%
AB 127 786 38.6%

45
Paste co-
brand logo
hereCost conclusions
• To recap our three desiderata:
1. Large sample size - maximise precision
2. High response rate – minimise non-response bias
3. Questionnaire length – maximise amount of data collected
• Condition BC (postcard reminder plus redesigned letters) is clear winner. For a fixed
budget:
1. Increases sample size by 2.6%
2. Increases response rate by 8.7 percentage points
3. No shortening of questionnaire required
• Above assumes increasing response rate will decrease non-response bias.
• Examine this assumption next

46
Paste co-
brand logo
here
Non-response bias

47
Paste co-
brand logo
here
Response rate improvement
efforts should be guided by some
knowledge of how groups likely
to be affected by the efforts relate
to key survey variables.
Bob Groves (2006)

48
Paste co-
brand logo
hereNon-response and non-response bias
The question
• Established that we can increase response rates using TDM interventions
• But, literature tells us that response rate often poorly correlated with non-
response bias (eg Groves and Peytcheva, 2008)
• Do our observed response rate increases reduce non-response bias?
Approach
• Cannot measure non-response bias directly for key survey variables
because don’t know true population distributions
• What we can do:
1. Compare demographic distribution of samples achieved in each condition with external
population data – shows absolute non-response bias for variables we are not
interested in (and which we can remove through weighting)
2. Compare distributions of samples across other survey variables – on the strong
assumption that bias can only decrease as response rate increases shows relative
non-response bias reductions

49
Paste co-
brand logo
here
Non-response bias: demographic
variables

50
Paste co-
brand logo
hereDemographic variables examined
• Variables
• Gender
• Age
• Ethnic group
• Work status
• Whether parent or legal guardian for child under 16
• Provides help for family members, friends, neighbours with long-
term physical or mental illness
• Sexual orientation
• Religion

51
Paste co-
brand logo
hereDemographic comparisons: results
• No meaningful differences between experimental groups
except for:
• age
• work status

52
Paste co-
brand logo
hereResults: age
Control A
Pre-
notification
B
Postcard
reminders
C
Enhanced
letters
D
Short
question-
naire
AB AC BC ABC ABCD
Response
rate 32.7% 32.9% 37.8% 34.4% 34.6% 38.6% 34.9% 41.5% 40.3% 43.6%
Age
group
Col. % Col. % Col. % Col. % Col. % Col. % Col. % Col. % Col. % Col. %
18-24 3.6% 2.9% 3.1% 4.9% 3.7% 3.2% 3.1% 3.8% 3.2% 2.6%
25-34 8.5% 8.7% 9.7% 7.5% 8.2% 9.0% 8.2% 9.4% 10.3% 9.8%
35-44 12.0% 13.7% 13.1% 13.2% 12.0% 13.2% 13.4% 14.6% 12.7% 14.2%
45-54 16.8% 16.3% 17.2% 17.2% 16.2% 17.4% 18.1% 19.0% 18.1% 19.9%
55-64 20.1% 19.0% 19.1% 21.6% 20.7% 20.2% 20.6% 20.4% 22.1% 19.7%
65-74 21.7% 22.5% 21.8% 18.5% 20.3% 21.0% 20.8% 18.7% 20.3% 18.2%
75-84 13.1% 13.4% 12.2% 12.9% 14.3% 12.1% 12.4% 10.7% 10.0% 12.3%
85+ 4.2% 3.5% 3.7% 4.0% 4.5% 4.0% 3.3% 3.4% 3.3% 3.3%
Base: 419,163
969 1,119 1,021 1,032 1,150 1,032 1,227 1,195 1,300

53
Paste co-
brand logo
hereAge distribution compared to population figures
ONS 2014
mid year
estimates
for England
Control ABCD
Response
rate
32.7% 43.6%
Age group Col. % Col. %
18-24 11.5% 3.3% 2.8%
25-34 17.4% 7.7% 8.6%
35-44 16.6% 11.5% 14.0%
45-54 17.9% 16.7% 18.9%
55-64 14.3% 20.2% 19.5%
65-74 12.1% 22.5% 19.6%
75-84 7.3% 13.6% 13.1%
85+ 3.0% 4.5% 3.5%
• Apply design weights to make comparison
• Comparator: Office for National Statistics population estimates for England
• Differences mostly attributable to non-response bias – but may also be
some non-coverage bias

54
Paste co-
brand logo
hereAge comparisons
• Increased response rates associated with small increases
in under 55 and decreases in 65+
• Gives slight improvement in age profile
• But improvements very small relative to residual non-
response bias

55
Paste co-
brand logo
hereResults: Work status
Control A
Pre-
notification
B
Postcard
reminders
C
Enhanced
letters
D
Short
question-
aire
AB AC BC ABC ABCD
Response rate
32.7% 32.9% 37.8% 34.4% 34.6% 38.6% 34.9% 41.5% 40.3% 43.6%
Full-time paid work (30
hours or more each
week) 32.9% 33.5% 35.7% 36.2% 32.7% 34.9% 34.8% 34.8% 36.6% 34.7%
Part-time paid work
(under 30 hours each
week) 13.3% 13.5% 14.1% 14.6% 13.1% 13.9% 12.2% 14.3% 14.2% 14.2%
Full-time education at
school, college or
university 1.4% 1.1% 0.9% 1.3% 1.4% 1.1% 1.8% 1.5% 1.2% 0.9%
Unemployed 3.7% 3.0% 4.0% 2.3% 3.1% 3.6% 4.4% 4.1% 4.6% 2.9%
Permanently sick or
disabled 4.5% 5.3% 4.6% 4.9% 4.8% 4.2% 4.5% 4.5% 4.2% 6.9%
Fully retired from work 36.4% 35.6% 32.8% 33.8% 36.6% 34.7% 34.2% 32.7% 32.1% 33.2%
Looking after the
home 5.4% 5.6% 5.4% 4.6% 6.0% 5.6% 5.9% 5.4% 4.3% 4.7%
Doing something else 2.4% 2.5% 2.4% 2.3% 2.3% 2.1% 2.1% 2.8% 2.7% 2.5%
Base: 405,899 936 1,087 974 939 1,115 999 1,188 1,155 1,179

56
Paste co-
brand logo
here
Employment rate compared with Annual Population
Survey (APS) rate
In employment
2014 Annual Population Survey (16+) 59.3%
Control 46.3%
A: Pre-notification 47.4%
B: Postcards 49.6%
C: Enhanced letters 51.3%
D: Short questionnaire 43.7%
AB 50.5%
AC 46.5%
BC 49.1%
ABC 52.6%
ABCD 48.5%
• Apply design weights to make comparison
• Not strictly comparable – GPPS: 18+; APS: 16+. Different survey
questioning protocols
• Despite this, real differences apparent: GPPS has too few in employment
• Increasing response rates generally increases employment rate a little

57
Paste co-
brand logo
hereWork status comparisons
• Increased response rates associated with small increases
in % in work and decreases in % retired
• Aligns with change in age profile
• Appears that bias slightly reduced
• Again appears to be large residual bias

58
Paste co-
brand logo
here
Non-response bias: key survey
variables

59
Paste co-
brand logo
hereKey survey variables
Q3. Generally, how easy is it to get through to someone at your GP surgery on
the phone?
Q4. How helpful do you find the receptionists at your GP surgery?
Q15. How convenient was the appointment you were able to get?
Q18. Overall, how would you describe your experience of making an
appointment?
Q22. Did you have confidence and trust in the GP you saw or spoke to?
Q24. Did you have confidence and trust in the nurse you saw or spoke to?
Q25. How satisfied are you with the hours that your GP surgery is open?
Q28. Overall, how would you describe your experience of your GP surgery?
Q29. Would you recommend your GP surgery to someone who has just moved to
your local area?
Q30. Do you have a long-standing health condition?
Q45. Overall, how would you describe your experience of NHS dental services?

60
Paste co-
brand logo
hereKey survey variables: results
• Very few significant differences – barely more than would
be expected by chance
• And no meaningful ones

61
Paste co-
brand logo
here
Conclusions

62
Paste co-
brand logo
hereConclusion
• For a fixed budget: adding a postcard reminder and
improving letter design increases sample size and response
rate without reducing questionnaire length
• However:
• Increased response rates make no tangible differences to
values of key survey variables
• Small improvements in age profile and employment rates, but
these minor relative to overall level of non-response bias
• Despite this we have a priori grounds for preferring higher
response rates
• Survey now uses postcard reminders and redesigned letters

63
Paste co-
brand logo
here
Paste co-
brand logo
here
Thank you
For further information, please contact:
Gerry.Nicolaas@ipsos.com
Patten.Smith@ipsos.com
05/09/2014
© Ipsos MORI This work was carried out in accordance with the requirements of the international quality standard for market research, ISO 20252:2006 and with the Ipsos MORI Terms and Conditions which can be found here

ESRA 2015 postal survey is dead, long live the postal survey

Recommended

Recommended

More Related Content

Similar to ESRA 2015 postal survey is dead, long live the postal survey

Similar to ESRA 2015 postal survey is dead, long live the postal survey (20)

Recently uploaded

Recently uploaded (20)

ESRA 2015 postal survey is dead, long live the postal survey

Editor's Notes