What one needs for succesfull survey designs

  • 136 views
Uploaded on

By Jannes Hartkamp

By Jannes Hartkamp

More in: Business , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
136
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. What One Needs for Successful Adaptive Survey Designs Jannes Hartkamp DESAN Research Solutions hartkamp@desan.nl
  • 2. DESAN Research Solutions • Amsterdam, The Netherlands, 1986 • large-scale socio-economic surveys • data collection, processing, digital reporting • survey software • survey modes – postal – face-to-face – phone – web – secondary data collection from registers, databases etc.NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 2
  • 3. Choosing modes • Four main survey modes – postal – face-to-face – phone – web • Single or multi-mode? • Concurrent or sequential or mixed mixed? • Which mode(s) for whom? • Differentiated (ASD) or same for all? • Choice for respondent? • Quality, costs, practicalities • What is needed to optimise the survey design?NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 3
  • 4. Mode checklist • survey subject – sensitivity – audiovisuals? – questionnaire length • availability of contact details – address, phone number, @ddress • survey population characteristics – age, ethnicity, education etc. • general response experiences and trends • response details from previous flowsNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 4
  • 5. Response in general • If contact details are known and correct… response costs speed multilingual face-to-face ++ -- - - telephone + - + + paper - + -- -- web -- ++ ++ ++ • … but phone numbers and e-mail @dresses are often unknown or incorrect – public telephone register coverage decreases steadily – varies highly by age, ethnicity and genderNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 5
  • 6. A case • Survey on sport and exercise in The Hague • Good example of what is needed for successful Adaptive Survey DesignNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 6
  • 7. The Hague Sport Survey • population of The Hague, age 4-70 (4-11 par) • subject: sports and other exercise + opinion • targets: – 4500 respondents – 8 districts, 8 age groups, 3 ethnic groups – special targets for immigrant youngsters, certain districts • sampling frame: population register • letter from dep. mayor to all citizens sampled • fieldwork January – March 2010NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 7
  • 8. Population characteristics • population details known from the register: – sex – age – ethnicity – district (address) • contact details: name, addressNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 8
  • 9. Response estimation sources • previous flows of the survey (DESAN, others) • other recent surveys in The Hague (DESAN) • other recent surveys in other major cities in order to estimate • telephone register coverage rate (TRCR) • response rates per mode by ethnicity and ageNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 9
  • 10. Some overall estimations • Estimated telephone response 20% - 40% BUT • estimated % of found phone numbers varied from 50% among non-immigrant Dutch to 11% (!) among Moroccan Dutch => phone alone is no option • estimated internet response varied from 3% to 7% => web alone is no option • estimated postal response varied from 5% to 13% => paper alone is no optionNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 10
  • 11. Some overall estimations • overall internet response: 5% • overall paper response: 9% • overall paper+web non-respondents telephone number rate: 33% • overall paper+web non-respondents telephone response rate: 31% • overall response: 5% + 9% + (33% * 31% * (100%-5%-9%)) = 23%NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 11
  • 12. Some detailed estimations Turkish Moroccan ‘Dutch’ Target n 393 283 2100 Response estimates: -internet (I) 3% 3% 6% -paper (P) 6% 5% 13% I+P non-response 91% 92% 81% phone number 19% 11% 50% rate phone response 30% 20% 40% rate -telephone (T) 5% 2% 16% TOTAL (S+I+T) 14% 10% 35% Sample needed 2770 2822 5966NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 12
  • 13. The Hague design • paper + web to all – concurrent • telephone among non-respondents – sequential • oversampling based on response rate and phone number estimates by ethnic group • (oversampling in certain ages, areas, ethnicities) • paper in Dutch, web and telephone scripts also in English and TurkishNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 13
  • 14. The Hague results • 4501 respondents (!) • total, ethnicities, districts targets reached • underperformance age 18-23 – many young adults who left home • incorrect addresses • unknown phone numbers • low response ratesNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 14
  • 15. Response by ethnic group 40% 36% 35% 35% 30% 30% 28% 25% 24% 22% 20% 15% 15% 15% 13% 10% 9% 10% 7% 5% 5% 6% 4% 5% 0% Moroccan/Marocain Turkish/Turque Dutch/Neerlandais TOTAL internet pap(i)er telephone TOTALNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 15
  • 16. Response by age 50% 45% 43% 42% 40% 35% 30% 31% 30% 30% 26% 25% 22% 24% 21% 20% 16% 15% 15% 10%11% 9% 9% 8% 10% 7% 5% 5% 5% 5% 0% 4-11 12-17 18-23 24-70 TOTAL internet paper telephone TOTALNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 16
  • 17. If blinded… …and only Moroccan and ‘Dutch’ Moroccan Dutch TOTALpopulation 500 4500 5000share in population 10% 90% 100%phone numbers found 14% 49% 46%telephone response rate 22% 35% 35%total response rate 3% 17% 16%total response n 15 772 787share in net sample (response) 2% 98% 100%NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 17
  • 18. The Hague conclusion • Combining administrative data with general and specific estimations on contact data and response rates yielded (probably) optimal modes mix, efficient sample. • Same design next time? Perhaps not. All estimations will have changed.NPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 18
  • 19. Finally Good example of what is needed for ASD: • population details • data to estimate – telephone register coverage rate – response rate => detailed meta/paradata from comparable previous surveys are crucial Bad example of ASD: • It was no ASD: differentiated oversampling, but same mode approach for all groupsNPSO 20-3-2012 The Hague Jannes Hartkamp – What One Needs for Successful Adaptive Survey Designs 19