Using flash type questions –
stroke of luck or curse for
data quality?
GOR 2009, Vienna




 Stefan Laufer, Jens Pietschma...
Our Research issues

  Survey results
  Are there differences between the flash sample and the non-flash
  sample with reg...
Methodical characteristics

  Survey topic: media and celebrities

  Sample provided by ODC Services.com (Munich)
   start...
Screenshot
Which TV-shows would you like to watch, if you could design your own TV-programme?




                        ...
Background: Definition ‚Flash‘ in our study


  Concentration on Flash-Question types which are included in EFS
  Survey i...
Comparison of survey results




                               6
Screenshot
Please tell us how likeable you find the celebrities you know.




                                            ...
Comparison Survey Results.
    Please tell us how likeable you find the celebrities you know.

                 Text      ...
Comparison Survey Results.
    Please tell us how likeable you find the celebrities you know.

                 Text      ...
Screenshot
How long have you watched the followíng tv-channels yesterday?




                                            ...
Comparison Survey Results.
How long have you watched the following tv-channels yesterday?

                               ...
Comparison Survey Results.
How long have you watched the following tv-channels yesterday?

                               ...
Screenshot
Please tell us how credible you find the media you know.




                                                  ...
Comparison Survey Results.
       Please tell us how credible you find the media you know.

                             T...
Comparison Survey Results.
       Please tell us how credible you find the media you know.

                             T...
Field Facts




              16
Field Facts.


                            Total             Text             Picture         Flash
 Started              ...
Field Facts.
Analysis of respondents (only flash)



                                       Suspended (22)            Comp...
Users‘ opinion




                 19
Users‘ opinion.
Diagnostic Items.




 Questions                                    Text   Picture           Flash
 The qu...
Users opinion.
Open answers (Original German)




         Endlich mal ne drag &       Die Machart war sehr
             d...
Summary
  In the survey results no systematic bias was found between the
  different groups

  Users like the new kind of ...
Summary




                                    Users Opinion
     Survey Results   Field Facts




                      ...
Thank you very much for your attention!



               Further information:
        http://globalpark.de/blog/gor09



...
Globalpark, headquartered near Cologne and with offices in London, New York and Vienna, is one of the
                   l...
Instructions for use




© 2008 – Globalpark AG, Globalpark UK Ltd., Globalpark USA, Globalpark Oesterreich GmbH


The inf...
Upcoming SlideShare
Loading in …5
×

Using flash type questions – stroke of luck or curse for data quality?

1,915 views
1,827 views

Published on

Online surveys are becoming more and more interactive. Respondents can use new flash question types. Card sorting, clicking on coloured and interactive buttons, ranking of items shown as pictures – the options seems to be nearly unlimited. Research institutes and panel providers expect more fun for the respondents while answering the surveys. But what’s about the results? Are there really effects on respondents and, much more important, is there any effect on the data quality of survey results? - A number of methodological questions emerge, one of them being associated with the measurement possibilities and usability.

Published in: Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,915
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Using flash type questions – stroke of luck or curse for data quality?

  1. 1. Using flash type questions – stroke of luck or curse for data quality? GOR 2009, Vienna Stefan Laufer, Jens Pietschmann 1
  2. 2. Our Research issues Survey results Are there differences between the flash sample and the non-flash sample with regard to data quality? Field Facts What about other factors like: completion rate answering time of questionnaire answer-length in open-ended-questions acceptance of flash between groups of respondents Users’ Opinon What do users say to the new flash version? Comparison of diagnostic items 2
  3. 3. Methodical characteristics Survey topic: media and celebrities Sample provided by ODC Services.com (Munich) startet: 901 / completed: 813 / completion rate: 87,8% Field period: 12.-17.3.2009 Gross sample quotet by age, gender and education One survey, monardic procedure with three versions of each question Routing by random to one of the three versions Text Picture Flash the „classic“ design of online surveys the „new“ design of online surveys 3
  4. 4. Screenshot Which TV-shows would you like to watch, if you could design your own TV-programme? 4
  5. 5. Background: Definition ‚Flash‘ in our study Concentration on Flash-Question types which are included in EFS Survey in software version 6.0 Following questions types have been used and analyzed: standard matrix open ended question drag & drop matrix 5
  6. 6. Comparison of survey results 6
  7. 7. Screenshot Please tell us how likeable you find the celebrities you know. 7
  8. 8. Comparison Survey Results. Please tell us how likeable you find the celebrities you know. Text Picture Flash 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 5 = very unlikeable n max. 261 4 = unlikeable 3 = neither likeable nor unlikeable n min. 5 2 = likeable 1 = very likeable 8
  9. 9. Comparison Survey Results. Please tell us how likeable you find the celebrities you know. Text Picture Flash 5 3 4 2,5 2,81 2,75 2,72 2 3 1,5 1 0,5 2 0 Text Picture Flash 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 5 = very unlikeable n max. 261 4 = unlikeable 3 = neither likeable nor unlikeable n min. 5 2 = likeable 1 = very likeable 9
  10. 10. Screenshot How long have you watched the followíng tv-channels yesterday? 10
  11. 11. Comparison Survey Results. How long have you watched the following tv-channels yesterday? more than 2 h 0,5 - 2 h 0,25 - 0,5 h haven't seen 3,9 20,2 38,5 37,4 Text 7,3 20,3 30,7 41,8 Picture 4,4 17,9 30,1 47,6 Flash 2,3 22,7 30,4 44,6 Text 4,3 19,5 27,7 48,4 Picture 2,2 16,7 32,2 48,9 Flash 7,1 26,7 24,7 41,6 Text 6,2 22,5 22,1 49,2 Picture 7,6 23,3 25,6 43,5 Flash 1,6 10 16,4 72 Text 1,25,4 17,4 76 Picture 2,3 9,1 20,1 68,5 Flash 3,2 11,7 85 Text 0,4 11 88,6 Picture 10,3 89,7 Flash Text 0,8 2,8 15,1 81,3 Picture 4,6 7,9 87,5 Flash 2,3 8,2 89,5 0% 20% 40% 60% 80% 100% n max. 261 n min. 219 11
  12. 12. Comparison Survey Results. How long have you watched the following tv-channels yesterday? more than 2 h 0,5 - 2 h 0,25 - 0,5 h haven't seen 3,9 20,2 38,5 37,4 Text 7,3 20,3 30,7 41,8 Picture 4 4,4 17,9 30,1 47,6 Flash 3,5 2,3 22,7 30,4 44,6 Text 3,59 3,55 3,52 4,3 19,5 27,7 48,4 Picture 3 2,2 16,7 32,2 48,9 Flash 2,5 7,1 26,7 24,7 41,6 Text 22,5 2 6,2 22,1 49,2 Picture 7,6 23,3 25,6 43,5 Flash 1,5 1,6 10 16,4 1 72 Text 1,25,4 17,4 76 Picture 0,5 2,3 9,1 20,1 68,5 Flash 0 3,2 11,7 85 Text Text 88,6Picture Flash 0,4 11 Picture 10,3 89,7 Flash Text 0,8 2,8 15,1 81,3 Picture 4,6 7,9 87,5 Flash 2,3 8,2 89,5 0% 20% 40% 60% 80% 100% n max. 261 n min. 219 12
  13. 13. Screenshot Please tell us how credible you find the media you know. 13
  14. 14. Comparison Survey Results. Please tell us how credible you find the media you know. Text Picture Flash 5 4 3 2 1 l ng e te rt a ss er r au ld Z au s TV iv z u ? ge ta nt al to it ha FA cu … os eu Bi ck pa i tu Bl ch rs ie ch G Bu Ta rn ss Fo sc pl ki H pe Sp Ze l ds ts pe e Ex da s or St Su ge un op e Sp n ch Ta rR D te n de ts et te eu W t ur ch dd kf su an Sü nd Fr la ch ts eu D 5 = very non credible N max. 227 4 = non credible 3 = neither credible nor noncredible N min. 53 2 = credible 1 = very credible 14
  15. 15. Comparison Survey Results. Please tell us how credible you find the media you know. Text Picture Flash 5 3 2,5 4 2 2,43 2,4 2,38 1,5 3 1 0,5 2 0 Text Picture Flash 1 l ng e te rt a ss er r au ld Z au s TV iv z u ? ge ta nt al to it ha FA cu … os eu Bi ck pa i tu Bl ch rs ie ch G Bu Ta rn ss Fo sc pl ki H pe Sp Ze l ds ts pe e Ex da s or St Su ge un op e Sp n ch Ta rR D te n de ts et te eu W t ur ch dd kf su an Sü nd Fr la ch ts eu D 5 = very non credible N max. 227 4 = non credible 3 = neither credible nor noncredible N min. 53 2 = credible 1 = very credible 15
  16. 16. Field Facts 16
  17. 17. Field Facts. Total Text Picture Flash Started 901 287 286 328 Suspended 88 (9,8%) 15 (5,2%) 20 (6,9%) 53 (16,1%) Completed 813 (90,2%) 272 (94,8%) 266 (93,1%) 275 (83,9%) Time to complete 14:20 minutes 12:30 minutes 13:06 minutes 16:31 minutes %-valid answers 97,87% 98,79% 99,41% 96,76% (not no answer) Length of open answers („Why do you like to have a dinner with...“) Ø like to have dinner 9,3 8,8 9,9 9,3 Ø don‘like to have 7,2 7,0 7,1 7,5 dinner 17
  18. 18. Field Facts. Analysis of respondents (only flash) Suspended (22) Completed (31) total 16,4% 83,6% under 18 years 40,0% 60,0% 18-24 years 8,8% 91,2% Age Groups 25-34 years 12,4% 87,6% (n=328) 35-44 years 10,7% 89,3% 45-55 years 13,8% 86,2% 55+years 20,7% 79,3% Gender male 15,2% 84,7% (n=328) female 18,0% 81,9% Data transmitted by ODC-Services 18
  19. 19. Users‘ opinion 19
  20. 20. Users‘ opinion. Diagnostic Items. Questions Text Picture Flash The questions have been clear and 1,46 1,31 1,37 comprehensible. 3,53 The loading of the pages was slow. 4,35 4,41 ** a) b) 3,32 The survey was longer than expected. 3,38 3,65 * b) 1,49 I liked the design of the survey. 1,83 1,51 *a) I would like to take part in such a survey 1,38 1,56 1,41 again. * a) It was a pleasure to take part in this 1,51 1,71 1,55 survey. * a) 1,51 The survey was interesting. 1,79 1,54 * a) a) Flash => Text 1=strongly agree b) Flash => Text + Bild 5 =strongly disagree 20
  21. 21. Users opinion. Open answers (Original German) Endlich mal ne drag & Die Machart war sehr drop Umfrage interessant. Mehr davon. Es macht Spaß, die Ich finde den Aufbau des Antworten in die Fragebogens mit Kästchen Antwortfelder zu ziehen. verschieben Klasse. . Tolle Idee, die Fragebogen Sehr schöne neue Idee... und Antwortmöglichkeiten nicht mehr nur das einmal ganz anders zu langweilige anklicken!!! gestalten. 21
  22. 22. Summary In the survey results no systematic bias was found between the different groups Users like the new kind of questions and perceive them as innovative and up-to-date New technology with old ‚problems‘: methodical consequences applied to the new flash-questions (of course!) Flash technology shows all aspects of ‚something new‘ to the users: some more technological problems and the respondents must learn how to use drag&drop Further research is needed: what happened there and what are the users doing? Why do they stop answering? 22
  23. 23. Summary Users Opinion Survey Results Field Facts 23
  24. 24. Thank you very much for your attention! Further information: http://globalpark.de/blog/gor09 24
  25. 25. Globalpark, headquartered near Cologne and with offices in London, New York and Vienna, is one of the leading international suppliers of online feedback software for market research, human resources, marketing and customer-relations management. Within the German-speaking world, Globalpark is market leader in these About Globalpark fields. More than 1,000 satisfied clients worldwide use Globalpark software. Over 350 of these clients are international groups, leading market research institutes and consulting firms. Globalpark’s customers include renowned companies such as Continental, Daimler, Deutsche Lufthansa, Deutsche Telekom, the GfK Group, Siemens, Warner Music and Wrigley. Globalpark USA Globalpark AG Globalpark UK Ltd. 405 Lexington Ave. Kalscheurener Str. 19a 5 Archie Street New York, NY 10174 50354 Huerth London SE1 3JT United States of America Germany (Headquarters) Great Britain Tel.: +1 888 299 9422 Tel.: +49 2233 7933 6 Tel.: +44 207 403 3900 Locations Globalpark Oesterreich GmbH Wassergasse 25 1030 Vienna Austria Tel.: +43 1715 0289 11 Matthias Mustermann Director Sales eHR Contact Globalpark AG Tel.: +49 2233 7933 777 Email: matthias.mustermann-tester@globalpark.com
  26. 26. Instructions for use © 2008 – Globalpark AG, Globalpark UK Ltd., Globalpark USA, Globalpark Oesterreich GmbH The information contained in this publication is the property of Globalpark AG and its subsidiaries. Passing on and reproduction of this publication or any parts of it is only permitted with the express written approval of Globalpark AG or its subsidiaries. This publication is issued without any guarantee whatsoever. Information contained herein can be changed without prior notification. In addition Globalpark accepts no guarantee as to the exactness or completeness of the information, text, graphics, links and other items in this publication. This publication is a preliminary version, which is not subject to any valid licence agreement or other agreements with Globalpark. The software products offered by Globalpark AG, its subsidiaries and/or its distributors may contain software components of other software manufacturers. All products may show country-specific differences. Globalpark, Enterprise Feedback Suite, EFS, EFS Survey, EFS Panel, EFS Employee and EFS Leadership, other Globalpark products and services mentioned in the text and their corresponding logos are trademarks and can be registered trademarks of Globalpark AG and/or its subsidiaries in Germany and other countries worldwide. All other names of products and services are trademarks of their respective companies.

×