Mixed mode surveys

Mixed-mode surveys: Do different modes produce different data? Evidence from the Greek Candidate Survey Ioannis Andreadis Aristotle University of Thessaloniki

Greek Candidate Survey 2007
Target population: PASOK and ND candidates
Mixed-mode survey (different data collection modes in sequence)
Web Survey (self-administered)
Nonresponse follow-up by (CATI) including a very limited number of FTF interviews conducted as CATI (no show cards)

To mix, or not to mix?
A survey conducted without spending a single Euro
Mixed-mode to reduce total survey error
Avoid coverage error
Avoid omissions (lists from political parties)
Avoid duplications and erroneous inclusions (PIN must be used to enter the survey)
Without introducing measurement error?

Mode Impact of CATI
Impact of Interviewer
Social desirability
Positive in interview
Acquiescence (tendency to agree)
Aural channel
Recency effect (the last or one of the last offered answer categories is chosen)

Web survey
Coverage: “Non-Internet households are older , have less education , and lower incomes ”.
Multitasking (Skype call, new email, etc)
Possible primacy effect?
Do the answers provided by respondents differ by mode?

Age, Education, Income
YoB: sig. difference (t=3.47, p=0.001)
Web average: 1955.4, CATI average: 1960
Education
No difference (circa 95% university in both groups)
Income
No data, but all of them can afford to pay for an internet connection

Outcome of election and party
Web: 160 responses RR: (20%-60%)
30% of the total population have been elected; only 20% of web survey participants
116 (72.5%) are PASOK candidates.
Why the ratio of elected MPs is smaller in the group of Web respondents? (discussion)
Why PASOK candidates are more than ND candidates? (discussion)

Matching (which variables to include?)
Age reported by 76 of 81 (CATI) and 104 of 160 (Web)
Web: break-offs (age one of the last questions of a long questionnaire); Research in other sources raised the numbers to 79 and 139. Web average age raised by 0.84 years (older people – more dropouts?)
On the other hand the impact of this difference (by 5.6 years) on the questions asked in CCS may not be worth the trouble
Exact matching without age (outcome and party)
Only 4 sub-classes
No need to drop cases

Chances to win X 2 = 14 . 6 p =0. 006 X 2 =5.8 p =0.218

Left-Right Self-placement X 2 =24. 6 P <0. 00 1 X 2 =2.4 p =0.661

Satisfaction with Democracy X 2 =6.0 p =0. 0 14 X 2 =1.4 p =0.234

Satisfaction with Democracy in EU X 2 =4.4 p =0. 0 37 X 2 =0.2 p =0.885

Our democracy needs serious reform X 2 =12 p =0. 0 07 X 2 =4.3 p =0.233

Income and wealth should be redistributed to ordinary people X 2 =13.4 p =0. 0 01 X 2 =3.5 p =0.177

Women should be free to decide on matters of abortion (direction) X 2 =6.4 p =0. 0 40 X 2 =4.5 p =0.107

Women should be free to decide on matters of abortion ( intension ) X 2 =15.8 p <0. 0 01 X 2 =11.1 p =0.004

Providing a stable network of social security prime goal ( intension ) X 2 =17.0 p <0. 0 01 X 2 =11.9 p =0.003

Findings contradict previous research on mode effects
It is reported that CATI is associated with extreme positive replies (strongly agree)
Reverse order of agree disagree in CCS?
Do Recency and Acquiescence drive CATI respondents to give the last offered answer category that matches their direction?

Conclusions
Web and CATI give similar results after matching Web respondents to CATI respondents
Some differences when (almost) all replies are located in the same direction and differences are due to intensity
An idea (for discussion) on how recency and acquiescence influences CATI respondents

Explanation of different ratios
Elected are more busy. Their ratio is still lower in 2009
Those who failed to elect want to review the situation and learn more about what they have done wrong or missed?
In the sample of 2009 ND candidates are slightly more than PASOK candidates. From this observation I conclude that the larger ratio of PASOK candidates in 2007 was partly due to different rates of elected MPs between the two political parties. The proportion of 2007 elected MPs was 37% in the group of ND candidates and 24% in the group of PASOK candidates. This picture was reversed in the 2009 elections.

The future of web surveys in Greece
Large increase in penetration of Internet use in Greece (Eurostat 2009) .
The percentage of Internet users increased from 18% in 2003 to 45%
The percentage of households with Internet access rose from 12% in 2002 to 38% in 2009
In 2009 33% of households have broadband Internet connection

Candidate Survey Objects
Campaigns
No differences between the two modes
Issues and Policies
A lot of differences
Most of them corrected by matching
Direction vs Intensity
Democracy and Representation
Some differences

Dillman et al (2009) Social Science Research
For example, in Treatment 2 (Form A), overall satisfaction was measured by interviewers reading from the most positive rating labels to the most negative rating labels, ‘‘where ‘5’ means extremely satisfied, and ‘1’ means not at all satisfied. . ..” In Treatment 3 (Form B) respondents heard the question with the most negative rating label first, as ‘‘where ‘1’ means not at all satisfied and ‘5’ means extremely satisfied. . .” You may use any number.
If a recency effect occurs we would expect that those responding in Treatment 2 ‘‘Not at all satisfied” (i.e., last category presented to the respondent) would be significantly different in comparison to the same response in Treatment 3. Likewise, those responding in Treatment 3 ‘‘Extremely satisfied” (i.e. last category presented to the respondent) would be larger than in the other treatment. This is not the case. A difference of means test shows that none of the differences are statistically significant and there is no consistent trend in either direction. All chi-square tests for differences were also insignificant. Because of the lack of differences, or ever slight trends, these results are not shown in the table. Also, because of the lack of differences we have combined Treatments 2 and 3 together for all other analyses reported in this paper.

Comment on the previous finding
Recency occurs when all five (or more categories) are read. If polar points are the only labeled points and respondent is requested to provide a number, the possibility of recency is small.

Proposed model for CATI 1/2
If order: strongly disagree – strongly agree
Respondent agrees
Recency towards strongly agree
Acquiescence towards agreement
Outcome: strongly agree, Conclusion: extreme responses
If order: strongly agree – strongly disagree
Respondent agrees
Recency towards strongly disagree
Outcome: agree, Conclusion: non extreme responses

Proposed model for CATI 2/2
If order: strongly disagree – strongly agree
Respondent disagrees
Recency towards strongly agree
Outcome: disagree – neither/nor (significant impact of acquiescence)
If order: strongly agree – strongly disagree
Respondent disagrees
Recency towards strongly disagree
Outcome: disagree or strongly disagree

Mixed mode surveys

by Ioannis Andreadis on Jul 19, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Mixed mode surveys — Presentation Transcript