Your SlideShare is downloading. ×
0
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Steinmetz Tijdens Aias09
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Steinmetz Tijdens Aias09

419

Published on

Presentation on the representativeness of online surveys, presented by Stephanie Steinmetz on a lunch meeting at AIAS, University of Amsterdam on October 1, 2009

Presentation on the representativeness of online surveys, presented by Stephanie Steinmetz on a lunch meeting at AIAS, University of Amsterdam on October 1, 2009

Published in: Education, Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
419
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Comparing different weighting procedures for volunteer online panels Stephanie Steinmetz and Kea Tijdens AIAS Lunch Seminar, 1. October 2009 erasmus studio
  • 2. Outline Background Sources of errors in ((volunteer) web) surveys Weighing - a solution? Example for the German and Dutch WageIndicator data Results Conclusion and Outlook
  • 3. Background Increasing importance of web surveys Germany: between 2000 und 2007 from 3% to 27% (ADM, 2007) Advantages time and cost reduction, interactivity, flexibility, ‘worldwide’ coverage, no interviewer influence Disadvantages Representativeness? To what degree are (volunteer) web survey results representative of the general public?
  • 4. Types of web surveys (see Couper, 2000) Sample selection is probability based = representative - intercept surveys, - online access panels, - mixed-mode surveys not probability based = representative? - entertainment surveys - self-selected web surveys - volunteer online panels
  • 5. Sources of error Combination of causes 1 (Non)Coverage: number of people having internet access + differences between the persons with and without internet access. 2 Sampling/Self-selection: no comprehensive list of Internet users to draw probability-based sample + people with specific characteristics participate in a volunteer online panel. 3 Non-response: Not all persons finish the questionnaire, people with specific characteristics might have a higher non- response. + measurement, processing and adjustment errors
  • 6. Weighting - a possible solution? Weighting is a mean to correct subsequently for systematic survey errors and to adjust the sample to the target population. Expectation disappearance of significant differences between web survey & random reference survey. ☺ = web survey data can be adjusted to be representative of general public. = persistence of differences due to other error sources, like measurement and processing errors
  • 7. Solution: Post-stratification weighting Aim: Adjustment for demographic under- and over- representations between sample and target population Method: %population (reference data) / %sample (web) = weighting coefficient Findings: Necessary but has a rather limited impact (Vehovar et al. 1999, Loosvelt and Sonck 2008) corrects for proportionality but not necessarily for representativeness of substantive answers
  • 8. ...but Previous research comparing web and traditional methods Significant differences can be observed for web respondents. They... – are more intensive users of the Internet, more technically-oriented (Bandilla et al. 2003; Vehovar et al. 1999) – have a larger social trust & a greater subjective control over their lives (Lenhart et al. 2003) – are more politically and socially active (Duffy et al. 2005)
  • 9. Solution: Propensity Score Adjustment (PSA) Origin: experimental studies (Rosenbaum & Rubin, 1983) Aim: to correct for differences due to the varying inclination to participate in web surveys (Harrison Interactive). Findings: Mixed (Taylor 2005; Bethlehem & Stoop 2007) - some differences disappeared by demographic weighting, - some only after additional PSA, and - others continued to exist or become even larger
  • 10. PSA - method (see Schonlau et al. 2009) Web and probability-based reference survey are combined in one data file Logistic regression of people’s probability to participate in the web survey given demographic and/or attitudinal variables estimation of PS Make distribution of these propensity scores similar for web survey and random sample = calculation of weight wpsi (1/ psi if W = 1 (in the web survey), and 1/(1-psi) if W = 0 (in the reference survey)) web survey and random sample do not differ significantly for selected variables included in the PS
  • 11. Example - the WageIndicator data Web surveys: German and Dutch WageIndicator data, year 2006, employees, age 16-75, cross monthly income 400€-10000€ (Dutch net hourly income) NGerman= 21914 NDutch = 8015 Reference surveys: Same restrictions Germany (GSOEP, 2006) N= 7993 Netherlands (OSA, 2006) N= 2019
  • 12. Selection bias - socio-demographics Germany Netherlands LS SOEP LW OSA 100 100 80 80 60 60 40 40 20 20 0 0 low low medium medium high high 16-34 35-44 45-75 16-34 35-44 45-75 women women men men sex education cohort sex education cohort
  • 13. Selection bias - Labour markert Germany Netherlands LS SO P E LW OSA 100 100 80 80 60 60 40 40 20 20 0 manual non full part below above 0 manual manual nonmanual full part below above occupation workingtime unemployment occupation working time unemployment
  • 14. Selection bias - satisfaction Germany Netherlands LS S EP O LW O A S 100 100 80 80 60 60 40 40 20 20 0 0 not satisfied satisfied not satisfied satisfied not satisfied satisfied not satisfied satisfied health satisfaction jobsatisfaction health satisfaction jobsatisfaction
  • 15. Summary Similarities: underrepresentation of women, people between 45 und 75, part-timers, persons from regions with high unemployment, unsatisfied people Differences: underrepresentation of DE: highly educated, manual workers NL: low and medium educated, non-manual workers Two possible solutions a) Post-stratification weighting b) PSA
  • 16. Weights A) 6 post-stratification weights: W1= gender (2), education (2) and cohort (2) W2= gender (2), education (2), cohort (2) and part time (2) W3= gender (2), education (2), cohort (2) and nonmanual (2) W4= gender (2), education (2), cohort (2), part time (2) and jobsat W5= gender (2), education (2), cohort (2), nonmanual (2) and jobsat W6= part(2) and jobsat(2) B) 4 PSA weights PS1 = treat women edu2 coh2 nonman part perm nojob logwagemo PS2 = treat women edu2 coh2 nonman part perm nojob logwagemo + healthsat PS3 = treat women edu2 coh2 nonman part perm nojob logwagemo + jobsat PS4 = treat women edu2 coh2 nonman part perm nojob logwagemo + healthsat jobsat
  • 17. Results: Germany – Mean income Diff Diff1 Diff2 Diff3 Diff4 Diff5 Diff6 PS1 PS2 PS3 PS4 PS4 PS3 PS2 PS1 Diff6 Diff5 Diff4 Diff3 Diff2 Diff1 Diff 0€ 50 € 100 € 150 € 200 € 250 € 300 € 350 €
  • 18. Results: Germany – distributions Diff DiffW2 DiffW6 DiffPS1 DiffPS2 30 20 10 0 men women 16-34 35-65+ lowmed medhigh manual nonman full part dissat sat -10 Gender Cohort Education Nonmanual Part-time Jobsat -20 -30
  • 19. Results: Germany - Income Regression
  • 20. Results: NL – mean income Diff Diff1 Diff2 Diff3 Diff4 Diff5 Diff6 PS1 PS2 PS3 PS4 PS4 PS3 PS2 PS1 Diff6 Diff5 Diff4 Diff3 Diff2 Diff1 Diff -0,6 € -0,5 € -0,4 € -0,3 € -0,2 € -0,1 € 0,0 € 0,1 € 0,2 €
  • 21. Results: NL - distributions 40 30 20 10 0 men women 16-34 35-65+ lowmed medhigh manual nonman full part dissat sat -10 Gender Cohort Education Nonmanual Part-time Jobsat -20 -30 -40 Diff DiffW1 DiffW2 DiffW3 DiffPS1 DiffPS3
  • 22. Results: Netherlands - Income Regression
  • 23. Conclusion Impact weighting Both weighting methods show no substantial impact Moreover - no consistency within weights - for some weights differences become larger (?!) - effect of weights differ between countries Weighting cannot improve representativeness of (volunteer) web surveys Problems - Reference surveys (also biased?), mode effects, unobservables (not measured)
  • 24. Discussion Possible solutions for representativeness: Improving weights through inclusion of more variables or advanced/mixed weighting procedures Only mixed-mode surveys (time and cost-reduction disappears) Non-representative use of web survey data (only for experiments or exploratory analysis) OR questioning the definition of representativeness (content vs. methodological) survey quality ≠ absolute
  • 25. Pa rtt im e Fu M llt al Pa im e rtt e 15 -2 im Ma 4 e le yr 0% 2% 4% 6% 8% 10% 12% 14% Fu Fe 15 hi llt im m -2 gh al 4 e e yr er Fu Fe 15 hi llt m -2 gh im al 4 e e yr er Fe 15- hi Pa m 24 gh rtt al er im e yr e 15 hi gh Pa M -2 rtt al 4 er e yr Fu ime 25 lo llt -4 w im Ma le 4 er e yr Pa Fe 45 lo m -6 w rtt im al 4 er e e yr M 45 lo Pa -6 w rtt al e 4 er 15 yr Fu ime -2 lo llt M 4 w im a yr er Pa e F le 1 m rtt em 5- id im 24 dl al e Fu e e yr Telepanel_NL_% _2002 llt Fe 25 lo im m -4 w e a 4 er Fe le 1 yr m lo WageIndicator_NL_% _2005 Pa 5- rtt al 24 w im e yr er 15 -2 lo Pa e M 4 w World Value Survey_NL_% _1999 al rtt e yr er Labour Force Survey_NL_% _2005 Fu ime 45 m llt M -6 id im al 4 dl e e yr Fu e F em 25- m llt im 44 id dl e al e yr e Fe 45 hi Pa m -6 gh rtt al e 4 er im 45 yr h Pa e M -6 ig rtt a 4 he im le 2 yr r e 5- m Fu M 44 id dl Pa lltim e 4 al yr e m rtt e 5- im M 64 idd le e al e yr F 15 hi Fu em -2 gh llt al 4 im e yr er Pa e 15 lo M -2 w rtt im al 4 e e yr er Pa Fe 15- m rtt id im m 24 dl e a yr e Pa Fe le 2 m rtt m 5- id dl im al e 44 e Fu e yr llt Fe 45- l im m 64 ow e a yr er Fu Fe le 4 hi llt m 5- gh im al 64 Pa e e yr er Fe 25 rtt m -4 lo im al 4 w e yr er Fe e 2 5- m Fu ma 44 idd le le Pa lltim 45 yr h rtt e -6 ig im M 4 he e a yr r Fe le 4 m Fu m 5- id 64 dl llt al e im e 2 yr e 5- Fu llt M 44 low er im ale yr Fu e M 25- hi gh ll al 44 er Pa tim e yr e 45 Representativness of surveys rtt M -6 lo im al 4 w e e er Fe 45 yr h Fu m -6 ig al 4 he llt im e yr r 2 m Fu e M 5-4 id llt 4 dl im al e y e e 25 r m M -4 id al e 4 dl e 25 yr h -4 ig 4 he yr r m id dl e
  • 26. Outlook Comparison: more countries Methods: - Combination of different weighting techniques (see Lee & Valliant, 2009) - Weighting with ‚better‘ reference survey and more webograhic variables (LISS panel= parallel survey, identical questionnaire + same mode)
  • 27. The end Thanks' for listening... ...questions ? ...comments and suggestions? contact: steinmetz@fsw.eur.nl erasmus studio

×