Online surveys are becoming more and more interactive. Respondents can use new flash question types. Card sorting, clicking on coloured and interactive buttons, ranking of items shown as pictures – the options seems to be nearly unlimited. Research institutes and panel providers expect more fun for the respondents while answering the surveys. But what’s about the results? Are there really effects on respondents and, much more important, is there any effect on the data quality of survey results? - A number of methodological questions emerge, one of them being associated with the measurement possibilities and usability.

1. Using flash type questions –
stroke of luck or curse for
data quality?
GOR 2009, Vienna
Stefan Laufer, Jens Pietschmann
1

2. Our Research issues
Survey results
Are there differences between the flash sample and the non-flash
sample with regard to data quality?
Field Facts
What about other factors like:
completion rate
answering time of questionnaire
answer-length in open-ended-questions
acceptance of flash between groups of respondents
Users’ Opinon
What do users say to the new flash version?
Comparison of diagnostic items
2

3. Methodical characteristics
Survey topic: media and celebrities
Sample provided by ODC Services.com (Munich)
startet: 901 / completed: 813 / completion rate: 87,8%
Field period: 12.-17.3.2009
Gross sample quotet by age, gender and education
One survey, monardic procedure with three versions of each
question
Routing by random to one of the three versions
Text Picture Flash
the „classic“ design of online surveys the „new“ design of online surveys
3

4. Screenshot
Which TV-shows would you like to watch, if you could design your own TV-programme?
4

5. Background: Definition ‚Flash‘ in our study
Concentration on Flash-Question types which are included in EFS
Survey in software version 6.0
Following questions types have been used and analyzed:
standard matrix
open ended question
drag & drop matrix
5

6. Comparison of survey results
6

7. Screenshot
Please tell us how likeable you find the celebrities you know.
7

8. Comparison Survey Results.
Please tell us how likeable you find the celebrities you know.
Text Picture Flash
5
4
3
2
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
5 = very unlikeable
n max. 261
4 = unlikeable
3 = neither likeable nor unlikeable n min. 5
2 = likeable
1 = very likeable
8

9. Comparison Survey Results.
Please tell us how likeable you find the celebrities you know.
Text Picture Flash
5
3
4
2,5
2,81 2,75
2,72
2
3 1,5
1
0,5
2
0
Text Picture Flash
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
5 = very unlikeable
n max. 261
4 = unlikeable
3 = neither likeable nor unlikeable n min. 5
2 = likeable
1 = very likeable
9

10. Screenshot
How long have you watched the followíng tv-channels yesterday?
10

11. Comparison Survey Results.
How long have you watched the following tv-channels yesterday?
more than 2 h 0,5 - 2 h 0,25 - 0,5 h haven't seen
3,9 20,2 38,5 37,4
Text
7,3 20,3 30,7 41,8
Picture
4,4 17,9 30,1 47,6
Flash
2,3 22,7 30,4 44,6
Text
4,3 19,5 27,7 48,4
Picture
2,2 16,7 32,2 48,9
Flash
7,1 26,7 24,7 41,6
Text
6,2 22,5 22,1 49,2
Picture
7,6 23,3 25,6 43,5
Flash
1,6 10 16,4 72
Text
1,25,4 17,4 76
Picture
2,3 9,1 20,1 68,5
Flash
3,2 11,7 85
Text
0,4 11 88,6
Picture
10,3 89,7
Flash
Text 0,8
2,8 15,1 81,3
Picture 4,6 7,9 87,5
Flash 2,3 8,2 89,5
0% 20% 40% 60% 80% 100%
n max. 261
n min. 219
11

12. Comparison Survey Results.
How long have you watched the following tv-channels yesterday?
more than 2 h 0,5 - 2 h 0,25 - 0,5 h haven't seen
3,9 20,2 38,5 37,4
Text
7,3 20,3 30,7 41,8
Picture
4
4,4 17,9 30,1 47,6
Flash
3,5
2,3 22,7 30,4 44,6
Text
3,59
3,55 3,52
4,3 19,5 27,7 48,4
Picture
3
2,2 16,7 32,2 48,9
Flash
2,5
7,1 26,7 24,7 41,6
Text
22,5 2
6,2 22,1 49,2
Picture
7,6 23,3 25,6 43,5
Flash
1,5
1,6 10 16,4 1 72
Text
1,25,4 17,4 76
Picture
0,5
2,3 9,1 20,1 68,5
Flash
0
3,2 11,7 85
Text
Text 88,6Picture Flash
0,4 11
Picture
10,3 89,7
Flash
Text 0,8
2,8 15,1 81,3
Picture 4,6 7,9 87,5
Flash 2,3 8,2 89,5
0% 20% 40% 60% 80% 100%
n max. 261
n min. 219
12

13. Screenshot
Please tell us how credible you find the media you know.
13

14. Comparison Survey Results.
Please tell us how credible you find the media you know.
Text Picture Flash
5
4
3
2
1
l
ng
e
te
rt
a
ss
er
r
au
ld
Z
au
s
TV
iv
z
u
?
ge
ta
nt
al
to
it
ha
FA
cu
…
os
eu
Bi
ck
pa
i tu
Bl
ch
rs
ie
ch
G
Bu
Ta
rn
ss
Fo
sc
pl
ki
H
pe
Sp
Ze
l
ds
ts
pe
e
Ex
da
s
or
St
Su
ge
un
op
e
Sp
n
ch
Ta
rR
D
te
n
de
ts
et
te
eu
W
t
ur
ch
dd
kf
su
an
Sü
nd
Fr
la
ch
ts
eu
D
5 = very non credible
N max. 227
4 = non credible
3 = neither credible nor noncredible
N min. 53
2 = credible
1 = very credible
14

15. Comparison Survey Results.
Please tell us how credible you find the media you know.
Text Picture Flash
5
3
2,5
4
2
2,43 2,4 2,38
1,5
3
1
0,5
2 0
Text Picture Flash
1
l
ng
e
te
rt
a
ss
er
r
au
ld
Z
au
s
TV
iv
z
u
?
ge
ta
nt
al
to
it
ha
FA
cu
…
os
eu
Bi
ck
pa
i tu
Bl
ch
rs
ie
ch
G
Bu
Ta
rn
ss
Fo
sc
pl
ki
H
pe
Sp
Ze
l
ds
ts
pe
e
Ex
da
s
or
St
Su
ge
un
op
e
Sp
n
ch
Ta
rR
D
te
n
de
ts
et
te
eu
W
t
ur
ch
dd
kf
su
an
Sü
nd
Fr
la
ch
ts
eu
D
5 = very non credible
N max. 227
4 = non credible
3 = neither credible nor noncredible
N min. 53
2 = credible
1 = very credible
15

16. Field Facts
16

17. Field Facts.
Total Text Picture Flash
Started 901 287 286 328
Suspended 88 (9,8%) 15 (5,2%) 20 (6,9%) 53 (16,1%)
Completed 813 (90,2%) 272 (94,8%) 266 (93,1%) 275 (83,9%)
Time to complete 14:20 minutes 12:30 minutes 13:06 minutes 16:31 minutes
%-valid answers
97,87% 98,79% 99,41% 96,76%
(not no answer)
Length of open answers („Why do you like to have a dinner with...“)
Ø like to have dinner 9,3 8,8 9,9 9,3
Ø don‘like to have 7,2 7,0 7,1 7,5
dinner
17

18. Field Facts.
Analysis of respondents (only flash)
Suspended (22) Completed (31)
total 16,4% 83,6%
under 18 years 40,0% 60,0%
18-24 years 8,8% 91,2%
Age Groups 25-34 years 12,4% 87,6%
(n=328)
35-44 years 10,7% 89,3%
45-55 years 13,8% 86,2%
55+years 20,7% 79,3%
Gender male 15,2% 84,7%
(n=328)
female 18,0% 81,9%
Data transmitted by ODC-Services
18

19. Users‘ opinion
19

20. Users‘ opinion.
Diagnostic Items.
Questions Text Picture Flash
The questions have been clear and
1,46 1,31 1,37
comprehensible.
3,53
The loading of the pages was slow. 4,35 4,41 ** a) b)
3,32
The survey was longer than expected. 3,38 3,65 * b)
1,49
I liked the design of the survey. 1,83 1,51 *a)
I would like to take part in such a survey 1,38
1,56 1,41
again. * a)
It was a pleasure to take part in this 1,51
1,71 1,55
survey. * a)
1,51
The survey was interesting. 1,79 1,54 * a)
a) Flash => Text
1=strongly agree
b) Flash => Text + Bild
5 =strongly disagree
20

21. Users opinion.
Open answers (Original German)
Endlich mal ne drag & Die Machart war sehr
drop Umfrage interessant. Mehr davon.
Es macht Spaß, die Ich finde den Aufbau des
Antworten in die Fragebogens mit Kästchen
Antwortfelder zu ziehen. verschieben Klasse. .
Tolle Idee, die Fragebogen
Sehr schöne neue Idee...
und Antwortmöglichkeiten
nicht mehr nur das
einmal ganz anders zu
langweilige anklicken!!!
gestalten.
21

22. Summary
In the survey results no systematic bias was found between the
different groups
Users like the new kind of questions and perceive them as
innovative and up-to-date
New technology with old ‚problems‘: methodical consequences
applied to the new flash-questions (of course!)
Flash technology shows all aspects of ‚something new‘ to the users:
some more technological problems and the respondents must learn
how to use drag&drop
Further research is needed: what happened there and what are the
users doing? Why do they stop answering?
22

23. Summary
Users Opinion
Survey Results Field Facts
23

24. Thank you very much for your attention!
Further information:
http://globalpark.de/blog/gor09
24

25. Globalpark, headquartered near Cologne and with offices in London, New York and Vienna, is one of the
leading international suppliers of online feedback software for market research, human resources, marketing
and customer-relations management. Within the German-speaking world, Globalpark is market leader in these
About Globalpark
fields. More than 1,000 satisfied clients worldwide use Globalpark software. Over 350 of these clients are
international groups, leading market research institutes and consulting firms. Globalpark’s customers include
renowned companies such as Continental, Daimler, Deutsche Lufthansa, Deutsche Telekom, the GfK Group,
Siemens, Warner Music and Wrigley.
Globalpark USA
Globalpark AG Globalpark UK Ltd.
405 Lexington Ave.
Kalscheurener Str. 19a 5 Archie Street
New York, NY 10174
50354 Huerth London SE1 3JT
United States of America
Germany (Headquarters) Great Britain
Tel.: +1 888 299 9422
Tel.: +49 2233 7933 6 Tel.: +44 207 403 3900
Locations
Globalpark Oesterreich GmbH
Wassergasse 25
1030 Vienna
Austria
Tel.: +43 1715 0289 11
Matthias Mustermann
Director Sales eHR
Contact
Globalpark AG
Tel.: +49 2233 7933 777
Email: matthias.mustermann-tester@globalpark.com

26. Instructions for use
© 2008 – Globalpark AG, Globalpark UK Ltd., Globalpark USA, Globalpark Oesterreich GmbH
The information contained in this publication is the property of Globalpark AG and its subsidiaries. Passing on and
reproduction of this publication or any parts of it is only permitted with the express written approval of Globalpark AG or its
subsidiaries.
This publication is issued without any guarantee whatsoever. Information contained herein can be changed without prior
notification. In addition Globalpark accepts no guarantee as to the exactness or completeness of the information, text,
graphics, links and other items in this publication. This publication is a preliminary version, which is not subject to any valid
licence agreement or other agreements with Globalpark.
The software products offered by Globalpark AG, its subsidiaries and/or its distributors may contain software components of
other software manufacturers. All products may show country-specific differences.
Globalpark, Enterprise Feedback Suite, EFS, EFS Survey, EFS Panel, EFS Employee and EFS Leadership, other
Globalpark products and services mentioned in the text and their corresponding logos are trademarks and can be registered
trademarks of Globalpark AG and/or its subsidiaries in Germany and other countries worldwide. All other names of products
and services are trademarks of their respective companies.