This document summarizes the results of a study testing a mixed-mode questionnaire for Finland's Labour Force Survey, including cognitive interviews and a pilot survey comparing computer-assisted web interviewing (CAWI) to telephone interviewing (CATI). Key findings include: 1) Questionnaire design improvements like layout, navigation bars, and "Don't know" options impacted usability. 2) The CAWI response rate was 30%, suggesting cost savings, but CAWI respondents had more "distorted" data. 3) No significant mode effects were found for employment status, but working hours questions showed mode effects between CAWI and CATI formats. Further testing is recommended to reduce biases between modes.

1. Mixed mode in LFS:
questionnaire design and
mode-effects
Liisa Larja, Statistics Finland
9th Workshop on Labour Force Survey Methodology
Rome, 15th
of May 2014

2. StatFin/EssNet web-project:
1. Designing LFS-CAWI questionnaire + iterative testing
with cognitive interviews (N = 22)
2. Pilot survey (N = 8 000), experimental design (”split-
plot”) with DK-option, working time and job search
questions. Comparison to CAWI data from the same
period. Paradata, respondent feedback.

3. Content of the presentation:
1. Questionnaire design: results & recommendations
2. Data collection process
3. Mode-effects: WSTATOR (employment status)
4. Mode-effects: HWACTUAL (actual working hours)
5. Qualitative feedback

4. 1. Questionnaire design: layout
 BlaiseIS default layout was not perceived as attractive
 Number of question / page
 Navigation bars
 Instructions
 Error messages
 Don’t know –option (DK)

5. Error message, DK
Instructions, info-button
Status bar
Number of questions, updating the
remaining questions
Scrolling
down for
navigation
bars

6. When the cursor points the info-
symbol, the instructions appear in a
box below.

7. Navigation bars:
Takaisin = Back
Jatka = Continue
Jatka myöhemmin =
Continue later/pause
Kyllä = Yes
En = No
(Don’t know not visible)

8. Kyllä = yes
Ei = no
En osaa sanoa = Don’t know
An error message: Please respond to this
question or choose the “Don’t know” option.
DK-option, version 1

9. Kyllä = yes
Ei = no
En osaa sanoa = Don’t
know
DK-option, version 2

10. 2. Data collection process
 Response rate 30 %  potential for savings in data
collection costs when the sample is large enough
 CAWI-data more ”distorted” ? CAPI non-response
more evenly distriputed with regard to educational
background, gender and age than CAWI-data.
 Speed: 40 % responses were received withing 3 days
 after that switch to CAPI?

11. 3. Mode-effects: WSTATOR (employment status)

12. 3. Mode-effects: WSTATOR (employment status)
No significant mode-effects, but….
• media attention remains if unemployment
rate 8,2 % instead of 7,4 %
• sample for pilot study was small
• CAWI respondents answer more often that
they have been looking for job
• = mode effect or selection bias?

13. 4. Mode-effects: HWUSUAL (actual working hours)
”One question” –format for HWACTUAL:
“How many hours did you
work in your main job
during the week 7.10.-
13.10.2013?”
___ hours ___ minutes

14. ”Grid” –format for HWACTUAL:
“How many hours did
you work in your main
job during the week
7.10.-13.10.2013?”
Date Hours Minutes
Monday 7.10.2013 ___ ___
Tuesday 8.10.2013 ___ ___
Wednesday 9.10.2013 ___ ___
Thursday 10.10.2013 ___ ___
Friday 11.10.2013 ___ ___
Saturday 12.10.2103 ___ ___
Sunday 13.10.29013 ___ ___
Total ___ ___

15. Average number of hours worked (CATI-data
and two different CAWI-datas)

16. 5. Qualitative feedback
 Too intimate information on employer
 Questions do not fit for people with multiple jobs
without main job or with scholarship
 All kind of entrepreneurial activity is not covered
 Working time items do not fit for people in atypical
employment
 Pensioners do not understand point of reference week

17. Conclusions and lessons learnt
 Response rate of 30 % and speed of data collection
encourage to continue towards mixed-mode LFS.
 Visual design is important: balanced layout, intuitive
navigation bars, informative error messages, a few questions per
page, hidden instructions, hidden DK.
 CAWI data more ”distorted”
 In working time questions different format for CATI
and CAWI may be needed to reduce mode-effects
(”functional equivalency”). Differences in WSTATOR
need to be further analysed.

18. Thank you!
Contact: liisa.larja@stat.fi