Workshop session 10 - Alternatives to CATI (3) address-based sampling and push-to-web
•
0 likes•170 views
Social Research Centre workshop - Telephone Surveying in the Post-Modern Era, held Thursday 10 October 2019. Presentation by Anna Lethborg - Research Director (Social Research Centre)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Workshop session 10 - Alternatives to CATI (3) address-based sampling and push-to-web
Similar to Workshop session 10 - Alternatives to CATI (3) address-based sampling and push-to-web (20)
More from The Social Research Centre
More from The Social Research Centre (7)
Recently uploaded
Recently uploaded (20)
Workshop session 10 - Alternatives to CATI (3) address-based sampling and push-to-web
- 1. A subsidiary of: Alternatives: address-based sampling (A-BS) and sequential mixed-mode Telephone Interviewing in the Post-Modern Era Conference Melbourne 10 October, 2019
- 2. www.srcentre.com.au Authors Anna Lethborg Research Director, Quantitative Research Consulting, Social Research Centre Dr Benjamin Phillips Senior Research Director, Survey Methodology, Social Research Centre 2
- 4. www.srcentre.com.au Three core design features Address-based sampling (A-BS) Mail and phone recruitment Sequential mixed-mode data collection 4 1 2 3
- 5. www.srcentre.com.au Address-based sampling A-BS is not new but is being increasingly used for large- scale social surveys internationally Dates to mid-2000s in the U.S. Landline RDD coverage was beginning to be recognised as a problem Telephone response rates were decreasing 5 • Near universal coverage of physical addresses • Geocoded data allows for accurate sub-national sampling • Able to append auxiliary data • Still a work in progress Rationale 1
- 6. www.srcentre.com.au Sample frame 6 Source: PSMA Australia (2018) 1 Geo-coded National Address File (G-NAF) Maintained by PSMA Australia (jointly owned by Commonwealth and State / Territory govts Over 13,000,000 addresses Updated quarterly Three reliable sources of address information Incl. validated addresses with Postal Address Flag (PAF) State and Territory Governments Australian Election Commission National Mailing Contributor 9% 7% 66% 3% 2%10% 3% 86% 9.2% unuseable 58.3% unuseable AES (2016)
- 7. www.srcentre.com.au Dual-modes of recruitment Primary recruitment mode is mail Invitation and reminder letters Reminder postcards Questionnaire 8 • Mail is an authoritative contact method • Inclusion of non-contingent incentives • Can send additional informational materials • Phone reminders may be more persuasive Rationale 2 Ability to match addresses to phone numbers (56.4%) also allows for telephone recruitment Reminder calls
- 8. www.srcentre.com.au Sequential mixed-mode Data collection method developed from late 2000s, publications from early 2010s Staggered approach 1. Push-to-web phase 2. Hard copy questionnaire sent 9 • Maximise online completions before introducing hard copy • Data quality • Reduced cost • High response rates Rationale 3 Non-responders sent 2-3 reminders before second mode introduced
- 9. www.srcentre.com.au Recruitment schedule 10 Online data collection (push to web) Mixed-mode data collection -7 days 0 days 7 days 19 days 47 days 54 days 66 days 87 days
- 10. www.srcentre.com.au Outer envelopes Authoritative sponsor No graphics / messages Personally addressed Letters Brief with white space Part of a select group Vary tone / message Support materials Brochure 1800 number / email Website Hard copy booklet Designed cover page Reply paid envelope Incentives $5 non- contingent $10 or $20 contingent Design considerations 11 “To the [SURNAME] household or current resident” “To the [SUBURB] resident” Branding
- 11. www.srcentre.com.au Branding examples 12 Invitation letter mailing Reminder letter Reminder postcard
- 12. www.srcentre.com.au Branding examples 13 Participant information on website Online survey (desktop) Hard copy questionnaire
- 14. www.srcentre.com.au How did we get here? 15 2019 Australian Election Study 2016 Australian Election Study 2018 World Values Survey 2015 Online Panels Benchmarking Study 2018 Australian Barometer Study
- 15. www.srcentre.com.au Evolving design 16 Length 45 mins / 24 pg 51 mins / 28 pg 48 mins / 28 pg 45 mins / 24 pg Incentives $10,000 prize draw $5 / $10 $5 / $10 $5 / $20 Pre-notification letter - - - 1 Reminder postcards 1 2 2 2 Reminder calls 1 2 2 2 Hard copy mailings 1 2 2 2 Reminder letters 3 2 2 2 Branding - Full Full Full Timeline Short push-2-web Extended Condensed Extended AES 2016 WVS 2018 Barometer 2018 AES 2019
- 16. www.srcentre.com.au 0% 5% 10% 15% 20% 25% 30% 35% -9 -5 -1 3 7 11 15 19 23 27 31 35 39 43 47 51 55 59 63 67 71 75 79 83 87 91 95 99 103 107 111 115 119 123 127 131 135 AES 2016 AES 2019 WVS 2018 Barometer 2018 Days elapsed Response rate comparison 17 30.7% 29.6% 27.2% 18.6% 21 32 43 47 AAPOR RR1
- 17. www.srcentre.com.au Respondent profile 19 Respondent characteristics Push-to- web Mixed- mode Total Aust. adults* Gender Male 42.3 42.1 42.2 49.2 Female 56.3 55.7 56.1 50.9 Age 18 to 24 3.6 2.4 3.1 12.2 25 to 34 11.2 8.9 10.2 19.3 35 to 44 13.3 14.2 13.7 17.1 45 to 54 17.5 13.8 15.9 16.7 55 to 64 20.8 20.0 20.5 14.9 65 to 74 21.6 20.0 20.9 11.3 75 and over 8.3 15.1 11.4 8.6 *Source: Australian Bureau of Statistics, Estimated Residential Population, December 2017 Available from http://www.abs.gov.au/ausstats/abs@.nsf/mf/3101.0
- 18. www.srcentre.com.au Respondent profile 20 Respondent characteristics Push-to- web Mixed- mode Total Aust. adults* Education University 43.7 35.2 39.8 25.6 Non-university 52.7 60.1 56.1 74.4 SEIFA Quintile 1 – Lowest 11.9 13.5 12.6 20.0 Quintile 2 20.4 21.5 20.9 20.0 Quintile 3 22.7 23.1 22.9 20.0 Quintile 4 21.8 20.6 21.2 20.0 Quintile 5 – Highest 23.3 21.3 22.4 20.0 *Source: Australian Bureau of Statistics, Estimated Residential Population, December 2017 Available from http://www.abs.gov.au/ausstats/abs@.nsf/mf/3101.0 Available in sample Possible to oversample lower quintiles Means lower RR
- 20. www.srcentre.com.au Operational considerations Field period is extensive (3-4 months) due to sequential design and need to cover multiple contact attempts Limited ability to change specifications once data collection started No ability to respond to deviations from production assumptions Hard copy questionnaire not changeable Pilot testing desirable Compared to dual-frame RDD, respondent profile is older, more female and better educated Postage contributes considerably to cost 22
- 21. www.srcentre.com.au Operational considerations For long questionnaires, consider two-phase approach: 1. Short screener to identify characteristics of interest (e.g. presence of children, health issues), collect household roster, alternate contact information (email, phone) 2. Recontact selected households: allows offline selection of individual from household roster, sampling of sub-populations, transition to alternative modes Considerations: Even longer data collection period Lower response rate than one phase Higher cost for incentives 23
- 22. www.srcentre.com.au Operational considerations Within-household selection not used since 2016 AES Believed to negatively impact response rate Respondents known to not follow sampling instructions Currently experimenting with next / last birthday 24 Male 48.9 42.9 49.1 Female 51.1 57.1 50.9 *Source: Australian Bureau of Statistics, Estimated Residential Population, December 2017 Available from http://www.abs.gov.au/ausstats/abs@.nsf/mf/3101.0 AES 2016 AES 2019 Aust adults* 18 to 34 years 10.9 13.2 31.5 35 to 44 years 11.2 13.7 17.1 45 to 54 years 15.7 15.9 16.7 55 years and over 62.2 57.2 34.8 Gender Age
- 23. www.srcentre.com.au Questionnaire considerations Limited complexity due to need for paper mode Limit skips, open-ended questions, etc. Certain question types will need a fundamental redesign Move to online / hard copy from phone likely to lead to mode effects Phone: response recency (more likely to select options read out last), response acquiescence and social desirability Online / hard copy: response primacy (more likely to select options appearing earlier), straight-lining and other response non-differentiation Efforts to understand mode effects will assist in transitioning 25
- 25. www.srcentre.com.au For us… 27 Unpack influences on response rate Non-response analysis Apply within-household selection routines Continued experimentation (e.g. registered post) Consider two-phase data collection NEXT STEPS Making sample-based corrections to improve representativeness
- 26. www.srcentre.com.au For you… If so, consider the following design elements: 28 Is A-BS sequential mixed-mode appropriate for your study? How important is response rate? Do you need to be able to sub- sample based on location? How long can you allow for data collection? What profile are you seeking? Is self-completion desirable? Is sequential mixed-mode required or is push-to-web sufficient for your needs? Have you prepared for the transition? Rotating phone response frames Consider bridging waves or fielding waves simultaneously
- 27. PO Box 13328 Law Courts Victoria 8010 03 9236 8500 A subsidiary of: 29 Thank you
- 28. www.srcentre.com.au References Battaglia, M, Link, MW, Frankel, MR, Osborn, L, and Mokdad, AH, 2008, ‘An evaluation of respondent selection methods for household mail surveys’, Public Opinion Quarterly, vol. 72, no. 3, pp. 459-469. Dillman, DA, 2017, ‘The promise and challenge of pushing respondents to the web in mixed-mode surveys’, Survey Methodology, vol. 43, no. 1, pp. 3-30. Dillman, DA, 2009, Internet, Mail, and Mixed-mode Surveys: The Tailored Design Method, 3rd Edition, pp. 234-299. Dykema, J, Jacques, K, Cyffka, K, Assad, N, Ganci Hammers, R, Elver, K, Malecki, KC and Stevenson, J, 2015, ‘Effects of sequential prepaid incentives and envelope messaging in mail surveys’, Public Opinion Quarterly, vol. 79, no. 4, pp. 906-931. Edwards, P, Roberts, I, DiGuiseppi, C, Wentz, R, Kwan, I, Cooper R, Felix LM and Pratap, S, 2009, ‘Methods to increase response to postal and electronic questionnaires (review)’, Cochrane Database of Systematic Reviews, issue 3, art. MR000008. 30
- 29. www.srcentre.com.au References Fan, W. & Yan, Z., 2010, ‘Factors affecting response rates of the web survey: A systemic review’, Computers in Human Behaviour, vol. 26, pp.132-139. Groves, RM, Cialdini, RB and Couper, MP, 1992, ‘Understanding the decision to participate in a survey’, Public Opinion Quarterly, vol. 56, no. 4, pp. 475-495. Iannacchione VG, Staab JM, & Redden DT (2003). Evaluating the use of residential mailing addresses in a metropolitan household survey. Public Opinion Quarterly 67(2):202–210. Link MW, Battaglia MP, Frankel MR, Osborn L, & Mokdad AH (2006). Address-based versus random-digit-dial surveys: comparison of key health and risk indicators. American Journal of Epidemiology 164(10):1019–1025. Tourangeau, R, 2017, ‘Mixing modes: tradeoffs among coverage, nonresponse, and measurement error’, in PP Biemer, E de Leeuw, S Eckman, B Edwards, F Kreuter, LE Lyberg, N Clyde Tucker and BT West (eds.), Total Survey Error in Practice, Hoboken, NJ, USA: Wiley, pp. 115-132. 31
Editor's Notes
- Welcome – thank you all for staying awake The next alternative for discussion is address-based sampling for sequential mixed-mode studies Apologies in advance to everyone from the ABS for my use of the abbreviation A-BS
- I’d like to thank Ben for his input into the slide development
- Another alternative to CATI is using address-based sampling to administer a sequential mixed-mode study, or what we call “A-BS sequential mixed-mode”. Before sharing some of our learnings since we first used this mode in 2015, I’ll introduce the key features of the design and tell you the rationale behind each
- There are actually three core elements to the A-BS sequential mixed-mode design Firstly we are using an address-based approach to sampling. For CATI studies we use landline or mobile telephone numbers, we’re talking about a list of postal addresses. Primary mode of contact is mail, we are also able to match phone numbers to the frame so you can do reminder calls And instead of telephone as the default mode for data collection, we use a two staged (sequential) approach to data collection. First, there is a push-to-web phase where respondents are invited to complete online and, following a period of time, a hard copy questionnaire is sent to provide an opportunity for the offline population to participate.
- Address-based sampling (A-BS) was developed in the mid-2000s in the U.S. (Iannacchione, Staab & Redden 2003; Link et al. 2006) as a response to the declining coverage of the landline random digit dialling (RDD) frame and declining telephone survey response rates.
- PSMA Australia (formerly Public Sector Mapping Authority) G-NAF sample frames select addresses from a database with near universal coverage (all physical addresses in Australia) of residential homes as distinct from the traditional block-listing approach. The G‑NAF was supplied by Mastersoft Group.
- Previous experience shows that very few completed questionnaires are obtained from non-validated addresses and that attempts to enumerate this sample also results in high rates of sample loss (58.3% unusable sample compared to 9.2% amongst the validated sample) and greatly reduced sample efficiency.
- Edwards 2009 cites evidence for sending a pre-notification letter
- Works with any sample with addresses; not limited to address based samples Sequential mixed mode design with paper not offered to begin with (Tourangeau, 2017) Questionnaire offered at 3rd or 4th contact (Dillman, 2017)
- 135 days of collection (4.5 months)!
- Match rate from AES 2019 Approach envelopes sponsored by an official / authoritative academic or government body (Dillman, 2017; Fan et al., 2010; Groves et al., 1992). Do not include graphics or external messaging on envelopes (Dykema et al., 2015) or brightly coloured envelopes as these can be mistaken for marketing materials (Dillman, 2009). Letters: Be brief and emphasise the importance / significance of the study (Dillman, 2009) Mention being part of a small selected group (Groves et al., 1992; Fan et al., 2010) Use different messages / tone / look used at different contact points (Dillman, 2009) Personally address survey invitations (Dillman, 2009; Fan & Yan, 2010). Use of an illustrated questionnaire booklet cover page (Edwards, et al., 2009) Include a token of appreciation (non-contingent incentive) with the survey request (Dillman, 2009)
- With thanks to ANU!
- Note: AES sample included non-validated addresses
- AAPOR RR1 AES 2016 (n=1,398) AES 2019 (n=1,211) WVS 2018 (n=1,211) Barometer 2018 (n=1,630) RR / % online 18.6% / 42.9% 30.2% / 60.7% 27.2% / 63.6% 30.7% / 78.1%
- Push-to web (n=662) Mixed-mode (n=549) Total (n=1,211)
- Push-to web (n=662) Mixed-mode (n=549) Total (n=1,211)
- Possible selection methods for address-based studies include a next / last birthday approach (Battaglia et al., 2008 and Olson, Stange & Smyth, 2014) AES 2016 n=1,398 AES 2019 n=1,211
- Prior to moving, rotate phone options to identify items affected by response recency (if you don’t already) Consider bridging wave or waves fielded simultaneously across both modes to determine effect of measurement on individual items