Nordic Statistical Meeting 26.-28.8.2019, Helsinki

1. 10/17/2019 Virtanen - Saari 2019 1
Like a box of chocolate? You never know what you’re gonna get -
Effects of a chocolate response incentive on data quality and
representativeness

2. • Since 2016 Statistics Finland has experimented with new types of response
incentives
• Direct monetary incentives have been consistently shown to increase response
rates better than gift certificate lottieries, but not always cost-effectively in every
population
• A cost-effective, easy to administer
10/17/2019 Virtanen - Saari 2019 2
Statistics Finland has experiment with
incentives to boost response

3. 10/17/2019 Virtanen - Saari 2019 3
Effective on populations otherwise
underrepresented in surveys
Ease of administration: Preferably
completely online
Cost-effective: Benefits should
outweight the costs
Characteristics of a good response incentive

4. • Studies have shown that almost everybody likes chocolate Rozin et al. 1991
• Such an incentive might be perceived more valuable than its intrinsic value (of ~2-3 euros)
• Handing out chocolate to respondents can be administered online by use of a single-use code. The
respondent may pick up his/her chocolate against a valid code from any R-kiosk location around the
country
10/17/2019 Virtanen - Saari 2019 4
Helpful R-kioski staff ready to distribute the incentive
Solution: A slab of chocolate which the respondent may redeem online
after responding online

5. 10/17/2019 Virtanen - Saari 2019 5
Research design
• This incentive experiment was conducted as part of the 2018 Finnish ISSP data
collection
• The ISSP 2018 (religion) was administered as web-push strategy designed to
boost early web response
• Sample was divided into two strata:
• Control group (1500), which was subjected to a lottery based incentive
• Experiment group (1700), which was subjected to both a lottery and redeemable chocolate
• The Experiment sub-sample was informed that they would get a bar of chocolate
in case they responded online within two weeks
• In addition all respondents were informed of a lottery of gift certificates among
survey participants

6. Our research objectives are threefold
10/17/2019 Virtanen - Saari 2019 6
Costs
Quality of
response
Response
rate
Does it make sense financially?
Better response rate
and a greater
proportion online?
Does this type of
incentive improve
the response quality
in terms of bias
reduction?

7. 10/17/2019 Virtanen - Saari 2019 7
Increase in the response rate? Success!
The incentive managed to increase the response rate by 4,6 percentage as well as
motivated more people to respond via online questionnaire.
Response rate in the control group 35,0 % (67,7 % of which online)
Response rate in the experiment group 39,6 % (76,0 % of which online)*
*Both increases found significant at 0.01 level

8. 10/17/2019 Virtanen - Saari 2019 8
Is the chocolate based motivation cost-effective?
Cost saving effects
- No need to send out paper
questionnaires to those who
have responded online
- No need to manually code
returned questionnaires of
web-respondents
- Less postal reminders due to
increase in response rate
Cost increasing effects
Each converted non-
respondent comes at a price
of round 3 euros per
redeemed chocolate bar +
people who would have
responded online anyway
- Expences related to general
administration of the incentive

9. 10/17/2019 Virtanen - Saari 2019 9
Costs related to mailing questionnaires
and manual coding were down 8,0
percent in the incentive group
The incentive and its
administration overall
however increased the total
cost per net response by 5,7
percent
What is important to notice is that these observed benefits scale up! With a bigger sample size we would
have broken even.

10. 10/17/2019 Virtanen - Saari 2019 10
• We calculated an index score of item non-response on three sets of
variables (social trust, religiousness, politics)
• Shows a Cronbach’s Alpha reliability of ~0.75 indicating that INR is systematic
for some respondents.
• An ANCOVA-test indicates that respondents motivated by chocolate
show no more IRN than control group (even when controlling for
respondent characteristics)
• Thus we conclude that there is no evidence on people rushing
through the questionnaire in hopes of quickly redeeming a bar of
chocolate ☺
Careless response style? (item non-response – INR)

11. 10/17/2019 Virtanen - Saari 2019 11
Respondent characteristics: Less bias?
We attempted to assess overall sample
representativeness using a response
propensity model
- We predicted response by sample
unit characteristics (age, sex,
education, region) in both sub-samples
Mixed results: Less bias in terms of
respondent age distribution but
somewhat more between educational
levels.
R-indicator indicated no substantial
difference within the sample
composition.
0 10 20 30 40 50
Men
Women
Total
Age group
15-24
25-34
35-44
45-54
55-64 *
65-74
No incentive Incentive

12. 10/17/2019 Virtanen - Saari 2019 12
Topic interest bias
• Less extreme views on religion observed in the chocolate sub-sample
indicating a decrease in bias related to topic interest (people more
interested in the survey topic tend to respond better)
• The survey includes a question on electoral participation – a known
distribution in the population (we know how many actually voted)
• The survey estimate was somewhat more accurate (significance p~0.1) in the
chocolate sub-sample

13. Thank you!
10/17/2019 Virtanen - Saari 2019 13