This document summarizes presentations from two sessions at the 2017 European Survey Research Association conference regarding encouraging survey participation through a "push-to-web" methodology. Key points discussed include the increasing but varied use of push-to-web designs, the importance of personalization and engagement across different contact modes, and priorities for further research such as optimal contact strategies and reducing burden when moving respondents online. Questions are also raised about pre-paid incentives, alternative modes, mobile phone surveys, and whether a single optimal push-to-web design may emerge.
2. 2Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Presentations
Session 1
1. Dr Don Dillman - The worldwide increase in use of web-push
methods that start with a mail contact; what have we learned
and where might we be going?
2. Dr Robert Neumann - Push2web or less is more? Experimental
evidence from a mixed-mode population survey at the
community level in Germany.
3. Sophie Nickson - Transformed respondent engagement
strategy; an overview of the research at ONS to develop
respondent materials.
4. Andrew Phelps - Quantitative testing of the most effective
advance communication strategies for a mixed mode (including
online) UK Labour Force Survey.
5. Patrice Mathieu - The 2016 Canadian Census: An Innovative
Wave Collection Methodology to Maximize Self-Response and
Internet Response.
Session 2
1. Dr Gudbjorg Andrea Jonsdottir - Pushing to web in the
ISSP
2. Dr Mónica Méndez - Notes from a Push2web(+mail)
survey: the ISSP 2014 Citizenship survey in Spain
3. Alice Fitzpatrick - Address-based Online Surveying
(ABOS): The impact of design features on response
4. Evangelia Kartsounidou - Pushing from telephone to
web: a low-cost and effective way to conduct national
election studies
3. 3Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Some emerging themes
Increasing use of
push-to-web survey designs
No single
magic bulletHighly variable
response rates
Mainly postal contact
but also telephone
4. 4Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
A framework for understanding response
to a push-to-web survey
Different stages of gaining cooperation
Participant perspective
Survey design features that can be manipulated
5. 5Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Focusing on
postal contact
6. 6Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Maximising push-to-web response rates
Motivated to
read the
mailing
Motivated to
go online
Motivated to
take part in
survey
Motivated to
open the
mailing
Motivated to
complete
questionnaire
Personalisation
Type of mailing;
e.g. envelope with
letter, postcard
Appearance of
mailing before
opening; e.g. logo
Personalisation
Easy to read; e.g.
length, font,
vocabulary
Appearance; e.g.
important,
professional
Clarity about the
purpose of the
mailing
Clarity about the
survey request
Use persuasive
reasons for taking
part
Clear instructions
for logging into
the questionnaire
Minimal effort
needed to enter
login details
Multiple access
methods; e.g. any
internet-enabled
device
Use multiple mailings:
• An optimum number of mailings
• An optimum length of time between mailings
• A diverse and yet coherent package of multiple mailings
Landing page
looks authentic
with clear
instructions
Design for mobile;
e.g. short, reduce
clutter and text
Avoid question
types that are
prone to break-
offs
Reduce cognitive
burden
7. 7Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
What about
telephone contact?
8. 8Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Maximising push-to-web response rates
Motivated to
engage with tel
interviewer
Motivated to
read email
& go online
Motivated to
take part
& give email
Motivated to
pick up phone
Motivated to
complete
questionnaire
Landing page
looks authentic
with clear
instructions
Design for mobile;
e.g. short, reduce
clutter and text
Avoid question
types that are
prone to break-
offs
Reduce cognitive
burden
9. 9Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Maximising push-to-web response rates
Motivated to
engage with tel
interviewer
Motivated to
read email
& go online
Motivated to
take part
& give email
Motivated to
pick up phone
Time of calling Personalisation
Maintaining
contact
Tailoring
Clarity about the
purpose of the
contact
Clarity about the
survey request
Use persuasive
reasons for taking
part
Email is
recognised as
genuine
Clear instructions
& easy to start
questionnaire
Multiple access
methods; e.g. any
internet-enabled
device
Multiple contact attempts:
• An optimum number & mix of phone calls & emails
• An optimum timing of phone calls & emails
• Refusal conversion
10. 10Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Fairly confident that the following has
a positive impact on P2W response
Personalisation
Engagement
– Tailoring & maintaining interaction with phone contact
– Multiple and varied contact attempts with postal contact
Communication style, complexity, length
Use of incentives
Reduce break-offs by avoiding the use of complex
question formats & cognitively difficult questions
Device-agnostic
Offer alternative mode to web non-respondents
11. 11Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Some doubt/uncertainty about
The type of incentives
The type and look of the mailings
Visual design of mailing content
The optimal package of multiple contact attempts
– Number of contact attempts, interval between contact
attempts, strategic approach to persuasive messaging,
mix of contact modes
How to reduce the effort of moving from offline
mode to online mode, particularly when relying on
postal contact
12. 12Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Suggested priorities for further research
Address-based samples
– How to encourage people to open the envelope when we don’t have a name?
– How to select respondent(s)?
Postal contact:
– What is the optimal strategy of multiple contacts?
– How to reduce the burden of moving from postal contact
to online questionnaire?
Sequential phone and email contact:
– How to encourage people by phone to provide email address?
– How to encourage people to open and read the email?
13. 13Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Questions for discussion
Pre-paid versus conditional incentives?
Alternative mode for web non-respondents
– Is it necessary for reducing non-response bias?
– For all or some web non-respondents?
Is mobile RDD+SMS an option in the near future?
Are we converging to a single optimal push-to-web design?
– Internet access will become (almost) universal
– Digital communication is becoming the norm for public & private services
– BUT the availability of sampling frames & contact information continues to vary by country
14. 14Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)
Thank you.
Editor's Notes
All postal except session 2 presentation 1 (various tests with p2w including some use of tel), and session 2 presentation 4 (RDD).
Strongly influenced by TDM approach; overlap with postal methods but the additional step of moving from letter to online.
The framework describes the response process temporally from the participant’s perspective, moving from opening and reading the postal contact, to understanding what is being requested and being persuaded to take part in the survey, to making the extra step from being persuaded by the postal contact to actually go online, and finally to making the effort to complete the web questionnaire. It presents an overview of survey design features at each of these stages which can be manipulated to improve response.
Adapted the framework for telephone –
Once contact is made, the tel interviewer has to quickly convince the listener that this is not a sales call but a genuine request to take part in something that is worth their time & effort – as for f2f, it is important to maintain contact and tailor the communication
Need to persuade them to provide an email address or provide login details over the phone which is prone to recording error
Although collecting email addresses over the phone can be problematic, once you have the email address you can send email invites with a direct link to the questionnaire which overcomes the barrier of moving from offline mode to online mode
Emails are often not opened and read – special attention for the subject heading and the sender’s address
Making telephone contact – the need to make multiple calls on different days, at different times
But if you have collected email addresses, then we are moving from a single offline contact mode to a sequential mixed contact modes and as demonstrated by Don Dillman using more than one mode of contact is better – need to use a mix of multiple telephone and email contacts, possibly going back to telephone contact to convert non-respondents
This framework for telephone & the previous one for postal can be used to identify what we already know about maximising online response rates
Adapted the framework for telephone –
Once contact is made, the tel interviewer has to quickly convince the listener that this is not a sales call but a genuine request to take part in something that is worth their time & effort – as for f2f, it is important to maintain contact and tailor the communication
Need to persuade them to provide an email address or provide login details over the phone which is prone to recording error
Although collecting email addresses over the phone can be problematic, once you have the email address you can send email invites with a direct link to the questionnaire which overcomes the barrier of moving from offline mode to online mode
Emails are often not opened and read – special attention for the subject heading and the sender’s address
Making telephone contact – the need to make multiple calls on different days, at different times
But if you have collected email addresses, then we are moving from a single offline contact mode to a sequential mixed contact modes and as demonstrated by Don Dillman using more than one mode of contact is better – need to use a mix of multiple telephone and email contacts, possibly going back to telephone contact to convert non-respondents
This framework for telephone & the previous one for postal can be used to identify what we already know about maximising online response rates
Ample evidence on positive impact of personalisation
Multiple & varied contact attempts:
Telephone contact – multiple calls at different times of the day and on different days of the week will increase the likelihood of making contact, additional calls for refusal conversion
If using phone contact, then tailoring and maintaining interaction are important
Postal contact – cooperation increases with the number of contacts, this is more effective when subsequent contact attempts look and feel different
Communication style, complexity & length: e.g. old Peter Lynn study showed that short informal letter was better than long formal letter; but Dillman would argue that it should still look professional (evidence?). Perhaps more evidence needed on visual design/look (see next slide).
Incentives – ample evidence; unconditional is better than conditional, monetary is better than non-monetary, response increases with the value but with diminishing returns; BUT unconditional incentives can be prohibitively expensive if the online response rate is relatively low; AND delivery of conditional incentives online can be cumbersome (in some countries when offering a choice of retailers)
Complex question formats will result in higher levels of break-off- e.g. traditional grids, ranking
Cognitively difficult questions will result in higher levels of break-off- e.g. recall of non-salient events over a long period, computations, reliance on looking up information from external sources (e.g. pay slips, bank account)
Smartphones are becoming the most important device for accessing the internet; for some groups the smartphone could be their only means of personal internet access – and this varies by country (e.g. smartphone access to the internet is particularly prevalent in developing countries)
Alternative mode for offliners and those who are online but haven’t made the effort to take the letter and go online
Across all traditional modes there is ample evidence that upfront incentives to all are better than incentives that are only given to those who take part – however, we are seeing mixed results for push-to-web surveys – Dillman, Spanish study, UK experience
Type & look of the mailings – evidence is mixed. For example, sometimes logos work and sometimes they don’t (possibly related to named samples versus address samples. Colour and size of envelopes may vary by country depending on country-specific conventions with regard to official mail and spam. Advance letters work for postal surveys but cost-effectiveness is less obvious for push-to-web surveys. A postcard thankyou/reminder works in postal surveys but generally not recommended to print usernames and passwords on postcards unless these can be folded. Our own experimental work using alternative mailing types such as pressure-sealed letters is not very promising.
Visual design of the content – how can we use visual design to encourage the recipient to read the letter and how can we ensure that the recipient reads the key messages even when they are only skim reading? At the same time, the visual design should not give the impression that the letter is marketing, sales, etc.
Although we know that multiple and varied contacts have a positive impact on response, we know less about how to design a strategy of using multiple carefully designed and timed contacts that reflect the characteristics of the survey and population, as well as tapping into different motivations for taking part in the survey.
Respondent selection instructions can be complicated and off-putting. It is easier to instruct any adult to go online but this will introduce selection bias. Evidence from the ESS and the UK community life survey show that (quasi) random selection such as last/next birthday and kish methods are ignored with about 25% of respondents not being the intended respondent. Asking all household members to complete the questionnaire when combined with incentives can result in respondent falsification. Asking “up to two adults” to complete the questionnaire exploits the fact that only a small proportion of households have more than two adults (at least in the UK) and reduces the risk of respondent falsification, but the instructions are non-intuitive and complicated which may put people off. We know of two studies (UK LFS & EU FRA) that are experimenting with asking any adult to go online and then asking this adult to provide information about household members so that a random selection can take place. The effectiveness of this approach needs to be demonstrated.
For both postal and telephone contact methods, no/limited experimental evidence on the best way of providing login instructions that are secure; e.g. username & password, or just a single password, length of username/password, the use of QR codes and SMS short codes, the number of steps needed to get to the first question.
Not sure about the last bullet point – probably mixed??? We know that short and simple questionnaires that use limited graphics etc are best. But there is ongoing research into the conversion of offline question formats into device-agnostic formats; e.g. traditional grids being changed into either collapsible grids or carousel grids. Perhaps this is more about the conversion of offline questionnaires into device-agnostic web questionnaires while maintaining measurement equivalence over time and across surveys???
