Mixed mode surveys


Published on

Presented at the fourth Conference of the European Survey Research Association (ESRA) held in Lausanne, Switzerland, from the 18th to the 22nd of July, 2011.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mixed mode surveys

  1. 1. Mixed-mode surveys: Do different modes produce different data? Evidence from the Greek Candidate Survey Ioannis Andreadis Aristotle University of Thessaloniki
  2. 2. Greek Candidate Survey 2007 <ul><li>Target population: PASOK and ND candidates </li></ul><ul><li>Mixed-mode survey (different data collection modes in sequence) </li></ul><ul><li>Web Survey (self-administered) </li></ul><ul><li>Nonresponse follow-up by (CATI) including a very limited number of FTF interviews conducted as CATI (no show cards) </li></ul>
  3. 3. To mix, or not to mix? <ul><li>A survey conducted without spending a single Euro </li></ul><ul><li>Mixed-mode to reduce total survey error </li></ul><ul><li>Avoid coverage error </li></ul><ul><ul><li>Avoid omissions (lists from political parties) </li></ul></ul><ul><ul><li>Avoid duplications and erroneous inclusions (PIN must be used to enter the survey) </li></ul></ul><ul><li>Without introducing measurement error? </li></ul>
  4. 4. Mode Impact of CATI <ul><li>Impact of Interviewer </li></ul><ul><ul><li>Social desirability </li></ul></ul><ul><ul><li>Positive in interview </li></ul></ul><ul><ul><li>Acquiescence (tendency to agree) </li></ul></ul><ul><li>Aural channel </li></ul><ul><ul><li>Recency effect (the last or one of the last offered answer categories is chosen) </li></ul></ul>
  5. 5. Web survey <ul><li>Coverage: “Non-Internet households are older , have less education , and lower incomes ”. </li></ul><ul><li>Multitasking (Skype call, new email, etc) </li></ul><ul><li>Possible primacy effect? </li></ul><ul><li>Do the answers provided by respondents differ by mode? </li></ul>
  6. 6. Age, Education, Income <ul><li>YoB: sig. difference (t=3.47, p=0.001) </li></ul><ul><ul><li>Web average: 1955.4, CATI average: 1960 </li></ul></ul><ul><li>Education </li></ul><ul><ul><li>No difference (circa 95% university in both groups) </li></ul></ul><ul><li>Income </li></ul><ul><ul><li>No data, but all of them can afford to pay for an internet connection </li></ul></ul>
  7. 7. Outcome of election and party <ul><li>Web: 160 responses RR: (20%-60%) </li></ul><ul><li>30% of the total population have been elected; only 20% of web survey participants </li></ul><ul><li>116 (72.5%) are PASOK candidates. </li></ul><ul><li>Why the ratio of elected MPs is smaller in the group of Web respondents? (discussion) </li></ul><ul><li>Why PASOK candidates are more than ND candidates? (discussion) </li></ul>
  8. 8. Matching (which variables to include?) <ul><li>Age reported by 76 of 81 (CATI) and 104 of 160 (Web) </li></ul><ul><ul><li>Web: break-offs (age one of the last questions of a long questionnaire); Research in other sources raised the numbers to 79 and 139. Web average age raised by 0.84 years (older people – more dropouts?) </li></ul></ul><ul><li>On the other hand the impact of this difference (by 5.6 years) on the questions asked in CCS may not be worth the trouble </li></ul><ul><li>Exact matching without age (outcome and party) </li></ul><ul><ul><li>Only 4 sub-classes </li></ul></ul><ul><ul><li>No need to drop cases </li></ul></ul>
  9. 9. Chances to win X 2 = 14 . 6 p =0. 006 X 2 =5.8 p =0.218
  10. 10. Left-Right Self-placement X 2 =24. 6 P <0. 00 1 X 2 =2.4 p =0.661
  11. 11. Satisfaction with Democracy X 2 =6.0 p =0. 0 14 X 2 =1.4 p =0.234
  12. 12. Satisfaction with Democracy in EU X 2 =4.4 p =0. 0 37 X 2 =0.2 p =0.885
  13. 13. Our democracy needs serious reform X 2 =12 p =0. 0 07 X 2 =4.3 p =0.233
  14. 14. Income and wealth should be redistributed to ordinary people X 2 =13.4 p =0. 0 01 X 2 =3.5 p =0.177
  15. 15. Women should be free to decide on matters of abortion (direction) X 2 =6.4 p =0. 0 40 X 2 =4.5 p =0.107
  16. 16. Women should be free to decide on matters of abortion ( intension ) X 2 =15.8 p <0. 0 01 X 2 =11.1 p =0.004
  17. 17. Providing a stable network of social security prime goal ( intension ) X 2 =17.0 p <0. 0 01 X 2 =11.9 p =0.003
  18. 18. Findings contradict previous research on mode effects <ul><li>It is reported that CATI is associated with extreme positive replies (strongly agree) </li></ul><ul><li>Reverse order of agree disagree in CCS? </li></ul><ul><li>Do Recency and Acquiescence drive CATI respondents to give the last offered answer category that matches their direction? </li></ul>
  19. 19. Conclusions <ul><li>Web and CATI give similar results after matching Web respondents to CATI respondents </li></ul><ul><li>Some differences when (almost) all replies are located in the same direction and differences are due to intensity </li></ul><ul><li>An idea (for discussion) on how recency and acquiescence influences CATI respondents </li></ul>
  20. 20. Explanation of different ratios <ul><li>Elected are more busy. Their ratio is still lower in 2009 </li></ul><ul><li>Those who failed to elect want to review the situation and learn more about what they have done wrong or missed? </li></ul><ul><li>In the sample of 2009 ND candidates are slightly more than PASOK candidates. From this observation I conclude that the larger ratio of PASOK candidates in 2007 was partly due to different rates of elected MPs between the two political parties. The proportion of 2007 elected MPs was 37% in the group of ND candidates and 24% in the group of PASOK candidates. This picture was reversed in the 2009 elections. </li></ul>
  21. 21. The future of web surveys in Greece <ul><li>Large increase in penetration of Internet use in Greece (Eurostat 2009) . </li></ul><ul><ul><li>The percentage of Internet users increased from 18% in 2003 to 45% </li></ul></ul><ul><ul><li>The percentage of households with Internet access rose from 12% in 2002 to 38% in 2009 </li></ul></ul><ul><ul><li>In 2009 33% of households have broadband Internet connection </li></ul></ul>
  22. 22. Candidate Survey Objects <ul><li>Campaigns </li></ul><ul><ul><li>No differences between the two modes </li></ul></ul><ul><li>Issues and Policies </li></ul><ul><ul><li>A lot of differences </li></ul></ul><ul><ul><li>Most of them corrected by matching </li></ul></ul><ul><ul><li>Direction vs Intensity </li></ul></ul><ul><li>Democracy and Representation </li></ul><ul><ul><li>Some differences </li></ul></ul>
  23. 23. Dillman et al (2009) Social Science Research <ul><li>For example, in Treatment 2 (Form A), overall satisfaction was measured by interviewers reading from the most positive rating labels to the most negative rating labels, ‘‘where ‘5’ means extremely satisfied, and ‘1’ means not at all satisfied. . ..” In Treatment 3 (Form B) respondents heard the question with the most negative rating label first, as ‘‘where ‘1’ means not at all satisfied and ‘5’ means extremely satisfied. . .” You may use any number. </li></ul><ul><li>If a recency effect occurs we would expect that those responding in Treatment 2 ‘‘Not at all satisfied” (i.e., last category presented to the respondent) would be significantly different in comparison to the same response in Treatment 3. Likewise, those responding in Treatment 3 ‘‘Extremely satisfied” (i.e. last category presented to the respondent) would be larger than in the other treatment. This is not the case. A difference of means test shows that none of the differences are statistically significant and there is no consistent trend in either direction. All chi-square tests for differences were also insignificant. Because of the lack of differences, or ever slight trends, these results are not shown in the table. Also, because of the lack of differences we have combined Treatments 2 and 3 together for all other analyses reported in this paper. </li></ul>
  24. 24. Comment on the previous finding <ul><li>Recency occurs when all five (or more categories) are read. If polar points are the only labeled points and respondent is requested to provide a number, the possibility of recency is small. </li></ul>
  25. 25. Proposed model for CATI 1/2 <ul><li>If order: strongly disagree – strongly agree </li></ul><ul><ul><li>Respondent agrees </li></ul></ul><ul><ul><li>Recency towards strongly agree </li></ul></ul><ul><ul><li>Acquiescence towards agreement </li></ul></ul><ul><li>Outcome: strongly agree, Conclusion: extreme responses </li></ul><ul><li>If order: strongly agree – strongly disagree </li></ul><ul><ul><li>Respondent agrees </li></ul></ul><ul><ul><li>Recency towards strongly disagree </li></ul></ul><ul><ul><li>Acquiescence towards agreement </li></ul></ul><ul><li>Outcome: agree, Conclusion: non extreme responses </li></ul>
  26. 26. Proposed model for CATI 2/2 <ul><li>If order: strongly disagree – strongly agree </li></ul><ul><ul><li>Respondent disagrees </li></ul></ul><ul><ul><li>Recency towards strongly agree </li></ul></ul><ul><ul><li>Acquiescence towards agreement </li></ul></ul><ul><li>Outcome: disagree – neither/nor (significant impact of acquiescence) </li></ul><ul><li>If order: strongly agree – strongly disagree </li></ul><ul><ul><li>Respondent disagrees </li></ul></ul><ul><ul><li>Recency towards strongly disagree </li></ul></ul><ul><ul><li>Acquiescence towards agreement </li></ul></ul><ul><li>Outcome: disagree or strongly disagree </li></ul>