Mastering Online Surveys


Published on

Presentation held at General Online Research conference 2009 (GOR09)

Published in: Business, Technology
1 Comment
  • Mastering Online Surveys through DIRECT link
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mastering Online Surveys

  1. 1. Agenda 1 Welcome and round of introduction Online surveys at a glance 2 - Trends and facts at your fingertips - 3 Practical Introduction Advanced features of online surveys 4 - Which tools to use when and how - Discussion 5 - Final Discussion and Exchange -
  2. 2. Trends towards in-house market research 2
  3. 3. Development of in-house market research Online surveys in companies up to 2003: Slow start Relatively limited range of applications up to 2003 (website checks, ad-hoc projects) The same method-focused approach as applied by institutes Used almost exclusively by market research departments Only a few major innovative projects are introduced by companies 3
  4. 4. Development of in-house market research Online surveys in companies from 2003: Fast, decentralized development Above-average growth in companies Internet diffusion Data basis Key to sales/communication Wide-ranging spectrum (marketing, HR, QM, development, services, etc.) High efficiency and effectiveness in individual projects Often insufficient integration into the company strategy; little operational transparency 4
  5. 5. Factors in the success of in-house online market research Broad level of Internet use in the consumption-relevant target groups (Source: ARD/ZDF study) 14 – 19 years: 96 % 20 – 29 years: 86 % 30 – 39 years: 77 % 40 – 49 years: 70 % 50 – 59 years: 58 % Decline in acceptance of telephone surveys (CATI) Decline in number of landlines Online surveys are more compatible with the new culture Increased demand for customer and market proximity (feedback culture) Continued improvements in corporate databases Web shop Customer clubs CRM and customer databases Boom in sampling panels (ciao, respondi, webfrager, eCircle, etc.) 5
  6. 6. Current trends in in-house online market research Above-average growth in in-house online market research Online customers/market research panels (ad hoc and longitudinal) Personalized and anonymous online surveys Automation of survey and analysis processes Broad area of application: Website checks, advertising impact, product tests, price research, customer satisfaction, QM, feedback Forrester Research quot;At least one-quarter of Fortune 100 companies will launch online customer communities this year (2008) in order to create higher levels of engagement with their customers and prospectsquot; quot;More and more large companies are moving market research in-house using do-it-yourself technologiesquot; 6
  7. 7. Online-Surveys 7
  8. 8. What happens when filling out the questionnaire? Fill out 1. page Saving answers Loading 2. page 2. page Fill out The given answers are saved by the server after every survey page and the next page is displayed 8
  9. 9. Access to online surveys Pop-ups/layers Survey Link on website E-mail invitation 9
  10. 10. Anonymous and personalized surveys Anonymous survey Online survey while the user is visiting a website • Inivitation to participate in a survey by pop-up window on entering or leaving the site Cookies prevent users from participating more than once Trend: Semi-automatic survey with trigger n-th visitor procedure Survey opens in a Survey is managed via pop-up quot;EFS Surveyquot; Anonymous user accesses the website 10
  11. 11. Access to anonymous surveys - EFS Survey project type: anonymous Popup Banner Layer Link on website 11
  12. 12. Anonymous and personalized surveys Personalized survey Importing address data and additional information in EFS survey Automatic generation of individual access codes Area of application: Customer surveys Higher return rate than for anonymous surveys Personal invitations by e-mail Survey is managed via quot;EFS Surveyquot; User data is imported into EFS Survey Personalized user performs survey 12
  13. 13. Access to personalized surveys - EFS Survey project type: personalized Link and password Personalized link 13
  14. 14. Type of projects anonymous personalized Implement popup window on Inviting participants by a website personalized e-mail Publish the survey link on a website Password is hidden in the link Publish the survey link in the User will be asked to enter the newsletter password 14
  15. 15. Online Surveys Facts at your fingertips 15
  16. 16. The basic outset Instrument Self-adminstered questionnaire No assistng interviewer Filling in and finishing the questionnaire based on the impression of the questionnaire Content Needs a clear structure Layout Questions and additional text Challenges Use of multi media elements (pictures, sounds, movies) Accessability (e.g. blind participants) Tech (bandwidth, screen resolution, Flash) 16
  17. 17. Advantages of online questionnaires Preparation No need to buy letters and stamps or hire interviewers No interviewer training necessary Can be created in short time In the field Quick feedback Mandatory questions Check answers for plausibility Automated sampling possible (quotas) Pictures, sounds or Movies can be displayed Realtime field report and statistics Data preparation Results data is readily available in the format desired Open answers are readily readable Tables and presentations can be produced directly 17
  18. 18. Possible errors in online survey (and surveys in general) Coverage Error Difference between the sample group and the sample population Nonresponse Error Certain subgroups of the sample didn‘t take part. Unit-Nonresponse / Item-Nonresponse Measurement Error Error caused by the instrument of measure (e.g. faulty questions) The measurement is incorrect (reliability) The wrong item has been measured (validity) Groves, Robert M. (1987): „Research on survey data quality“. In: Public Opinion Quarterly 51, 156-172 18
  19. 19. Answer behavior based on the questionnaire as a whole Basic approach of Web Usability Engineering Users don’t want to scroll Participants want to scroll. They want to see the whole page at once. Users don’t want to read Participants most likely only take notice of highlighted text elements. Of the non-highlighted text the upper left has the highest probability of being read. Most other text will be ignored Text from paper surveys should be cut by at least 50%. 19
  20. 20. Considerations When creating an online survey one has to consider The 10 Golden Rules Not so apt internet users Java-Script avaiable Flash available Screen resolution Slow connection issues (e.g. in Russia) 20
  21. 21. The Ten Golden Rules (1-5) Keep it simple (avoid foreign words, technical terms, initials or slang) Keep it short (avoid long sentences or sentence compounds) Keep it tangible (avoid abstract terms) Keep it honest (avoid leading questions) Keep it neutral (avoid controversial or loaded terms) 21
  22. 22. The Ten Golden Rules (6-10) Keep it real (do not you hypothetical questions) Keep it focussed (adress one item at the time, avoid „and“, „or“,“because“, etc.) Keep it clear (avoid double negations) Keep it manageable (do not demand too much from the participants) Keep it balanced (try to use an equal amount of positive and negative answers) 22
  23. 23. Question types Open ended questions excluded One can differentiate between 5 basic question types: Single response questions Multi response questions Matrix questions Drop-down questions Others (mostly combinations or variations of the above) 23
  24. 24. Question types: Single response question Single response question: • Straight forward question type • Uses radio buttons • Mind that it can’t be deselected! • Easy to realize and answer • Too many identical questions will become boring (Loop) 24
  25. 25. Question types: Multi response questions Multi reponse question: • Similar to singlepunch question • Allows for multiple answers • Can be deselected • Issue of the residual category (“None of the above”) 25
  26. 26. Question types: Matrix question Matrix question: • More efficient then several singlepunch questions • A lot of input on one screen • Randomization is advised • Limited number of statements is advised • Might become frustrating if mandatory • Might become boring if asked a lot in a row 26
  27. 27. Question types: Drop-down question Drop-down question: • Basically a variation of single response question • Unique to online questionnaires • Needs a “Please select” category • Essential: Not all answer options are displayed at once • Ideal for long answerlists • Best attention grabber when used as mandatory question (also see Dr. Mirta Galesic et al.’s work Eye-Tracking) 27
  28. 28. Question types: Variations E.g. Image questions: • Variation of the single/multi response question • Instead of buttons one clicks on pictures 28
  29. 29. Question types: Combination E.g. Drop-down Matrix: • Combination of Drop-down and Matrix question • Even more input then Matrix question (3 dimensions) 29
  30. 30. Practical introduction The first survey project - Sequence of operations - 30
  31. 31. Process sequence: Online survey in 7 steps 1. Create project 2. Enter questions 3. Activate project 4. Test project 5. Invite participants 6. Control progress 7. Export data 31
  32. 32. Question editor: Working on 3 levels Level 1: Level 2: Level 3: Structure view Page view* Question view (page list) Question 3: Family status Page 1: Personal data Project: Soziodemographic question survey Question 1: Question text: Page 1: Gender Please enter your Personal data questions family status. Question 2: Page 2: Age Employment status Answer 1 not married Question 3: Page 3: Answer 2 Family status Income status married Question 4: Page 4: Answer 3 Nationality Household structure divorced/widowed *1 page = 1 survey screen 32
  33. 33. Access to administration area Administrator creates projects in the password-protected login area: Username and password must be entered to login. 33
  34. 34. Screen structure 1 3 Navigation bar with modules Menu of the current module Breadcrumb / path indication Content 2 4 1 2 3 4 34
  35. 35. Online Surveys Advanced Features 35
  36. 36. Advanced features of an online survey Online survey offer many additional features to the fray which cannot normally be used in all other survey variants: URL/GET-Parameters and/or participants administration Mandatory questions Plausibility checks Randomization Dynamics in the questionnaire through routing, placeholders, dynamic lists 36
  37. 37. GET-/URL-parameters GET-/URL-parameter GET/URL-Parameter • Are one way to relay data about the participant to the results data prior to the start of the questionnaire • Commonly used for anonymous projects • They are essential when dealing with panel providers • E.g. If you are recruiting from several different sites and you want to differentiate from which site each participant came from 37
  38. 38. Participants administration Participants administration • Commonly used in personalized projects • A participants administration should ideally be able to contain all the additional variables which are needed for routing and data presentation for each participant. Thus making redundant questions unnecessary. • As well an panelist administration should enable you to invite and remind 38 all or selected participants.
  39. 39. Mandatory questions Mandatory question • Enable you to ensure that certain or all questions are filled in. • Can but should not be used with open ended questions • Might significantly lower the response rate or the response quality • Involvement of the participants can counter this effect (e.g. employee survey) • Incentives will make the response rate to stay but will lower the quality 39
  40. 40. Plausibility Checks Plausibility Checks • Enable you to ensure that certain or all questions are filled in correctly. • Here one should be especially cautious to test because an error here might ruin the whole study • Avoid checking to complex issues with these checks. Rather simplify them. 40
  41. 41. Exclusive Checkbox Exclusive Checkbox • Special solution to solve the issue of residual categories in multi response questions • Requires Java- Script • If available it is preferable to checking with a plausibility By clicking => deselect the check. selection Activate in answer options 41
  42. 42. Randomization of Items Randomization • Randomization of Items significantly lowers order effects • Additional functionalities like Grouping can allow for the participant to have an overview while still being shown a randomized list of items • Another approach is the “flip” which simply mirrors scales. 42
  43. 43. Filter/ Routing • Filters and Routing are one of the major advantages of online questionnaires compared to paper&pencil. • The participants will simply not see the pages they are not supposed to see. • Again, testing is crucial at this point. As with a faulty filter data will be inevitably lost. 43
  44. 44. Placeholder Placeholders are a simple and straight forward way to have a dynamic questiontext. Thus sparing the need to create multiple questions (and variables) just to have adifferent wording 44
  45. 45. Dynamic questionnaire structure (lists) Dynamics • As a more sophisticated version of filters further dynamics can be applied. • E.g. the answers to one questions determine the answer-options in the following question • One should be aware of avoiding participants seeing empty pages 45
  46. 46. Dynamic questionnaire structure (loops) Loops • A Loop is a function that allows to automatically generate multiple repeating questions from the answers to a source question. • The particular risk here is that with too many questions asked in a row either the response quality will decrease or the drop-out rate will increase or both. 46
  47. 47. Further possibilities • Recoding of data in the questionniare. • Quotas and Screen-outs • Dia-show • Prefilling Form Elements • Sending out mails from the questionnaire (e.g. “Tell a Friend“) • Questionnaire receipt 47
  48. 48. Working with the platform Setting up Working with Field phase Analysis Follow-up the platform the platform Enter or import participants at department level Create personalized invitation for participants Define printing and shipping points for paper-based questionnaires 48
  49. 49. Automation / CRM / transaction-based feedback 49
  50. 50. Challenge: Transaction-based feedback Transaction - Sales Company Customer - Consulting Feedback? Feedback is a prerequisite for optimizing products, services, advertising campaigns, support services, etc. If feedback is not requested, customers are likely to tell friends what they think and/or post information in the Internet (forums, blogs, ratings). People are ten times more likely to talk about negative experiences than • positive ones! 50
  51. 51. Standard process for automated online surveys Trigger 1 e.g. transaction CRM / ERP 2 integration Data management: Survey 3 and participants Report distribution Invitation 4 8 Report Field phase 5 Reporting 7 6 Real-time check 51
  52. 52. Interface to customer database CRM / ERP CRM / ERP EFS interface Web Services CRM / CC database manages customer transactions Simple connection via EFS Web Services Conculding transaction = ticket/event closed Relevant customer data is transferred automatically 52
  53. 53. Participant management & e-mail invitations Participant management • Export random sample • Import to participant management in EFS • Generate customised survey codes & URLs • Generate personal e-mails with title, first name, last name and URL • Take into account panelists who have been filtered out or who are no longer registered Integrated e-mail tool E-mail is either sent to the customer immediately or at a time specified 53
  54. 54. Field phase High-end solution: EFS Survey • CI-compliant layout • Password/code protection • SSL security • Plausibility checks during data entry • Personalization of the questionnaire • Complex questionnaire with a high level of quality 54
  55. 55. Automatic crisis management Process chain for negative assessments An e-mail is sent to a E-mail sales employee or the Trigger person responsible for the process Customer complaint • Customer number • Process • Trigger • etc. An action is triggered if values in the questionnaire exceed or fall short of specified limits (e.g. an infomail is sent to the service employee responsible)‫‏‬ 55
  56. 56. Reporting Field report Online reporting center Return checks Reporting platform with separate Time-dependent log-in for employees and customers access All the functions of a field report Termination rate/page-dependent Report downloads termination Survey frequencies & times Participants who have been filtered out Automatic reports Data export Results reports for decision-makers Export in all common data formats Index calculation Interface to evaluation software Download in reporting center 56
  57. 57. Example: After-sales feedback Automated customer survey 1 B2B platform Challenge: Increase customer loyalty Implementation: 3 2 1. Customer buys a product 2. Transaction is reported automatically to EFS Survey 3. EFS Survey automatically sends questionnaires tailored to the transaction (product, customer) to customers. Customer provides online feedback. Customer care management 4. Reporting to customer care management, PM or 4 similar 5. Specific data is transferred automatically to the CRM system 5 Benefit: Increases customer loyalty quot;Early-warning systemquot; for customer satisfaction Information for quality and innovation management CRM system 57
  58. 58. Example: Call center feedback 1 Automated customer survey Customer care Challenge: Increase customer loyalty Implementation: 1. Customer uses online or telephone support 4 2 service. Customer center solves the problem. 2. The CRM system records the process 6 3. The CRM system reports the process CRM automatically to EFS Survey system 3 4. EFS Survey automatically invites customers to take part in an online survey; consumer provides feedback 5 5. Reporting sent to the customer care management 6. Results data is transferred automatically Customer care Benefit: management Increase in customer loyalty secures sales through: Optimization of the service quality quot;Early-warning systemquot; for customer 58 satisfaction
  59. 59. Feedback triggers continuous improvement Collect / evaluate data Continuous optimization process Check / analyze Introduce measures 59
  60. 60. The Unipark program Support program for university research institutes and students Feedback channel for academic users Over 300 cooperation projects with university chairs Europe U.S. Canada Asia 60