Vovici Vision 2011: New Zen and the Art of Questionnaire Design


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • http://www.imdb.com/title/tt0095159/quotes
  • http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  • http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  • http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  • http://blog.vovici.com/blog/bid/18225/Long-Surveys-Turn-Respondents-into-Liars
  • http://blog.vovici.com/blog/bid/18156/Satisficing-and-Survey-Respondent-Behavior
  • http://blog.vovici.com/blog/bid/57499/Understand-Respondent-Behavior-to-Write-Better-Surveys
  • Egocentric bias: recalling the past in a self-serving manner, e.g., remembering one's exam grades as being better than they were, or remembering a caught fish as bigger than it really wasLeveling and Sharpening >: memory distortions introduced by the loss of details in a recollection over time, often concurrent with sharpening or selective recollection of certain details that take on exaggerated significance in relation to the details or aspects of the experience lost through leveling. Both biases may be reinforced over time, and by repeated recollection or re-telling of a memory.Telescoping effect > >: the tendency to displace recent events backward in time and remote events forward in time, so that recent events appear more remote, and remote events, more recent.</li>
  • http://blog.vovici.com/blog/bid/21978/Acquiescence-Bias-Agree-Disagree-Scale-Best-Practices
  • http://blog.vovici.com/blog/bid/25082/Social-Desirability-Bias-Sex-Drugs-Rocking-Surveys
  • http://blog.vovici.com/blog/bid/52933/Mode-Effects-Same-Survey-Different-Medium-Different-Results
  • http://blog.vovici.com/blog/bid/28597/Halo-Error-Lose-the-Halo-to-Gain-More-Accurate-Insightshttp://blog.vovici.com/blog/bid/31148/Response-Substitution-Answering-the-Unasked-Question
  • http://blog.vovici.com/blog/bid/18186/Good-Surveys-start-with-Good-Goals
  • http://blog.vovici.com/blog/bid/18186/Good-Surveys-start-with-Good-Goals
  • http://blog.vovici.com/blog/bid/18285/Order-Questions-Logically
  • http://blog.vovici.com/Blog/bid/18203/Screening-Questions-May-Indicate-Need-for-Better-Survey-Profiles
  • http://blog.vovici.com/Blog/bid/18095/Open-Ended-Questions
  • http://blog.vovici.com/blog/bid/21917/How-to-Maximize-the-Number-of-Open-Ended-Responses
  • http://blog.vovici.com/Blog/bid/18190/Skip-Logic-Conditional-Branches-in-Surveys
  • http://blog.vovici.com/blog/bid/28235/Sequential-vs-Grouped-Placement-of-Filter-Questions
  • http://blog.vovici.com/Blog/bid/18176/Demographic-Questions-Sample-Survey-Templatehttp://blog.vovici.com/Blog/bid/18047/Firmographic-Definition-Template
  • http://blog.vovici.com/blog/bid/27344/Respondent-Satisfaction-Post-Survey-Assessments
  • http://blog.vovici.com/blog/bid/18197/Types-of-Questions-Four-Building-Blocks-for-Constructing-Questionnaires
  • http://blog.vovici.com/blog/bid/18088/Open-Ended-Questions-vs-Closed-Ended-Questions
  • http://blog.vovici.com/blog/bid/18088/Open-Ended-Questions-vs-Closed-Ended-Questions
  • http://blog.vovici.com/blog/bid/18123/Yes-No-Questions-Common-Pitfalls-to-Avoid-When-Writing-Questionnaires
  • http://blog.vovici.com/blog/bid/18170/Multiple-Answer-Questions-Select-All-That-Apply-Best-Practices
  • http://blog.vovici.com/blog/bid/18201/Matrix-Questions-Powerful-but-Perilous-Technique
  • http://blog.vovici.com/blog/bid/26649/Bullet-Time-for-Matrix-Questions
  • http://blog.vovici.com/blog/bid/18228/Ranking-Questions-vs-Rating-Questions
  • http://blog.vovici.com/blog/bid/18228/Ranking-Questions-vs-Rating-Questions
  • http://blog.vovici.com/blog/bid/57326/Constant-Sum-Allocation-Questions-Best-Practices
  • http://blog.vovici.com/blog/bid/18230/Writing-Objective-Survey-Questions
  • http://blog.vovici.com/blog/bid/18230/Writing-Objective-Survey-Questions
  • http://blog.vovici.com/blog/bid/56281/The-Shoulds-Should-Nots-of-Question-Wording
  • http://blog.vovici.com/blog/bid/56281/The-Shoulds-Should-Nots-of-Question-Wording
  • http://blog.vovici.com/blog/bid/18842/Survey-Test-Mode
  • http://blog.vovici.com/blog/bid/18101/Favorite-Color-Survey-Results-How-Short-Choice-Lists-Lead-to-Wrong-Answers
  • http://blog.vovici.com/blog/bid/18101/Favorite-Color-Survey-Results-How-Short-Choice-Lists-Lead-to-Wrong-Answers
  • http://blog.vovici.com/blog/bid/18076/Rating-Scale-Labels-Label-End-Points-or-Every-Point
  • http://blog.vovici.com/blog/bid/21978/Acquiescence-Bias-Agree-Disagree-Scale-Best-Practices
  • http://blog.vovici.com/blog/bid/18620/Custom-Scale-Development
  • http://blog.vovici.com/blog/bid/18261/Common-Scales-to-Use-when-Writing-Questions
  • http://blog.vovici.com/blog/bid/18620/Custom-Scale-Development
  • http://blog.vovici.com/blog/bid/18083/When-to-Use-Drop-down-Lists-and-When-Not-To
  • http://blog.vovici.com/blog/bid/20397/No-Opinion-as-a-Question-Choice
  • http://blog.vovici.com/blog/bid/18133/Recommended-Survey-Length
  • http://relevantinsights.com/long-surveys
  • http://blog.vovici.com/blog/bid/17976/The-Long-and-the-Short-of-Questionnaire-Length
  • http://blog.vovici.com/blog/bid/50909/How-To-Shorten-a-Questionnaire
  • http://blog.vovici.com/blog/bid/17976/The-Long-and-the-Short-of-Questionnaire-Lengthhttp://blog.vovici.com/blog/bid/23110/Perceived-Questionnaire-Length
  • http://blog.vovici.com/blog/bid/18174/Closing-the-Feedback-Loop-Sharing-Results-with-Online-Community-Members-Respondents
  • http://blog.vovici.com/blog/bid/18667/Email-Trigger-a-Key-Aspect-of-EFM
  • http://blog.vovici.com/blog/bid/18053/Survey-Alerts-Trigger-Emails-Improve-Satisfaction
  • Vovici Vision 2011: New Zen and the Art of Questionnaire Design

    1. 1. Zen and the Art of Questionnaire DesignVovici VisionMay 18, 2011
    2. 2. Our Agenda: The Eightfold Path to Survey Enlightenment1. Right Perspective2. Right Intention3. Right Order4. Right Attention5. Right Speech6. Right Choice7. Right Length8. Right Action 2
    3. 3. More Questionnaire Design, Less Zen “Now let me correct you on a couple of things, OK? “Aristotle was not Belgian. “The central message of Buddhism is not ‘Every man for himself.’ “And the London Underground is not a political movement.” Not Vovici specific For other aspects, see 7 Habits of Highly Successful Surveys 3
    4. 4. The Eightfold Path to Survey Enlightenment Right Perspective: The Respondent as Human 4
    5. 5. Asking a Lot of the RespondentLiterally and Figuratively1. Interpret the meaning of a question2. Recall all relevant facts related to question3. Internally summarize those facts4. Report summary judgment accurately 5
    6. 6. Asking a Lot of the RespondentLiterally and Figuratively1. Interpret the meaning of a question2. Recall all relevant facts related to question3. Internally summarize those facts4. Report summary judgment accurately 6
    7. 7. Respondent Coping StrategiesDifferent Types of Processing 1. Interpret question meaning 2. Recall relevant facts 3. Internally summarize facts 4. Report summary judgment 7
    8. 8. Respondent Behavior Degrades as Survey Lengthens Optimizing Weak Satisficing Strong Satisficing Cheating 8
    9. 9. Weak Satisficing Strong Satisficing Selecting the first choice that Endorsing the status quo instead appears reasonable of change Agreeing with assertions Failing to differentiate in ratings (“acquiescence response bias”) Selecting “Don’t know” rather than giving an opinion Randomly choosingSource: Krosnick, J. A. (1991). “ResponseStrategies for Coping with the CognitiveDemands of Attitude Measures in Surveys.”Applied Cognitive Psychology, 5, 213-236. 9
    10. 10. Other Signs Respondents are Human Cognitive Social Survey Behaviors Behaviors Behaviors Satisficing Acquiescence bias Response styles Memory biases Social desirability bias Response substitution Economic behavior Halo error Mode effects Practice effects Panel conditioning 10
    11. 11. Memory Biases Choice-supportive bias Generation effect Osborn effect Source Confusion Change bias Illusion-of-truth effect Part-list cueing effect Spacing effect Childhood amnesia Lag effect Peak-end effect Stereotypical bias Consistency bias Leveling and Sharpening Persistence Suffix effect Context effect Levels-of-processing effect Picture superiority effect Suggestibility Cross-race effect List-length effect Positivity effect Telescoping effect Cryptomnesia Misinformation effect Primacy effect Testing effect Egocentric bias Misattribution Processing difficulty effect Tip of the tongue Fading affect bias Modality effect Reminiscence bump Verbatim effect Hindsight bias Mood congruent memory bias Rosy retrospection Von Restorff effect Humor effect Next-in-line effect Self-relevance effect Zeigarnik effect Source: http://en.wikipedia.org/wiki/Memory_bias 11
    12. 12. Acquiescence Bias Some respondents are simply agreeable, and indicate agreement out of politeness Other respondents expect that the researchers agree with the listed items and defer to their judgment Most respondents find agreeing takes less effort than carefully weighing each optional level of disagreement and agreement Source: Saris, Krosnick and Shaeffer, 2005 12
    13. 13. Social Desirability Bias, by Topic Health initiatives - Respondents exaggerate frequency of exercise and compliance with medical regimens Voting behavior - Respondents exaggerate intent to vote Illegal behavior - Respondents underreport drug usage and criminal history Sexual behavior - Respondents deny, sanitize or mainstream aspects of their sexual lives Bigotry - Respondents downplay any prejudices Salary - Poor respondents overstate income; rich respondents understate it 13
    14. 14. Mode Effects Face-to-face: a “guest” script Social desirability bias highest: Phone interviews: a “solicitor” 1. Telephone surveys script 2. Face-to-face surveys IVR interviews: a “voice mail” 3. IVR surveys script 4. Mail surveys Internet surveys: a “web form” script 5. Web surveys Mail surveys: a “form” script 14
    15. 15. Answers Patterns for Common Response Styles Completely Somewhat Neither agree Somewhat Completely Disagree Agree Response Style disagree disagree nor disagree agree agree Optimal Responding Extreme Response Style (ERS) Truncated Scales Response Range (RR) Mild Response Style Midpoint Response (MPR) Acquiescence Social/Anti-Social Styles Response Style (ARS) Disacquiescence Response Style (DARS) Socially Desirable Responding (SDR) Noncontingent Responding (NCR) 15
    16. 16. Response Styles by Country: Informed by Culture Source: Johnson, Kulesa, Cho, Shavitt, 2003; Vovici 16
    17. 17. Halo Error & Response Substitution • What makes cheesesteaks and Tastykakes® taste even better? • Fans like the food 11% more when the Eagles win. Victory tastes delicious. • Halo error confuses the true strengths and weaknesses of products and services. • Makes benchmarking attributes across competing brands and products unreliable (brands are well documented as introducing halo effects). • Leads to misinterpretation of satisfaction attributes. • Response substitution is when respondents answers to questions might sometimes reflect attitudes that respondents want to convey, but that the researcher has not asked about 17
    18. 18. What Respondents Like About Taking Surveys Nothing 34% Dislike surveys 24%Chance to voice their opinion 13% Earn an incentive 11% Like to be helpful 10% Interesting topics 9% Answers make a difference 7% 0% 20% 40% 60% 80% 100% Source: Vovici Survey Nation study, N = 100 RDD sampled U.S. adults 18
    19. 19. The Eightfold Path Right Intention: Worthy Purpose for Survey 19
    20. 20. The Second Factor: Worthy Purpose For a SurveyFocus on a Goal Be precise about what information you need to gather and what you plan on doing with it If your organization hasn’t done a survey in a while, the tendency is for every department to chime in with questions they want you to ask A narrow goal will help you to relentlessly simplify the survey "In the case of archery, the hitter and the hit are no longer two opposing objects, but one reality." - Zen in the Art of Archery 20
    21. 21. Is a Survey the Right Arrow to Hit the Target? Sometimes the best survey is to not do a survey at all Talk to stakeholders who will use the data to understand their wants and needs Is someone elsewhere in the organization doing a survey on this topic or researching this issue? Are customers (or employees or …) the only source of this information? "Two hands clap and there is Do your CRM, web analytics or other systems a sound. What is the sound hold data that would address this issue? of one hand?" - Hakuin Ekaku 21
    22. 22. Examples of GoalsGood Goals Bad Goals Determine how often callers into It’s been a while; we should do a help desk check the customer- survey. service web site before calling What are our customers thinking Prioritize the feature ideas for a right now? new product with prospects Maybe customers would like it if we Find out how satisfied customers provided support after 6 pm. are with tech support Let’s create a committee and see Determine if employees feel that what they want to find out from a organization is living up its core survey. values 22
    23. 23. The Eightfold Path Right Order: Questions’ Proper Sequence 23
    24. 24. The Third Factor: Questions’ Proper Sequence Screener Use the inverted pyramid approach, Open-Ended Questions drilling down General Questions Ask harder questions first, before respondents Specific Questions grow tired of the survey Demographics Follow-up 24
    25. 25. The Third Factor: Questions’ Proper Sequence Screener Conventionally, screeners route people out of survey Open-Ended Questions depending on answers to initial question General Questions Build panel profiles that Specific Questions including the screening Demographics information you need You have their attention; skip Follow-up them out to another survey Otherwise they get the impression their feedback isn’t valuable 25
    26. 26. The Third Factor: Questions’ Proper Sequence Screener Capture their views in their own words before biasing Open-Ended Questions them with your later questions General Questions o “What, if anything, do you Specific Questions like about…?” Demographics o “What, if anything, do you dislike…” Follow-up Answers provide color commentary to later closed- ended questions Answers validate choice lists 26
    27. 27. Trade Off Increased Abandonment for More Verbatim Responses Survey Concluding with Survey Beginning with Open-Ends Open-EndsAbandonment rate 1% 6%Responses with at least 61% 90%one verbatim answeredAverage length of 13 words 13 wordsverbatim answerSample size 70 79 Source: Vovici research on research, 9-24-09 27
    28. 28. The Fourth Habit: Order Questions Logically Screener Use Skip Use Patterns Branching Open-Ended Questions General Questions Specific Questions Demographics Follow-up 28
    29. 29. Sequential Filtering Provides Better Conversational Flow Sequential Filtering Grouped Filtering Source: Lisa Carley- Baxter, Andy Peytchev and Michele C. Black, 2010 29
    30. 30. The Third Factor: Questions’ Proper Sequence Screener Use demographic and firmographic questions to Open-Ended Questions profile respondents and their organizations General Questions Enables you to cross-tabulate Specific Questions & compare subgroups Demographics Pre-populate from CRM systems where possible Follow-up Place near end as tedious and intrusive but can be answered on “autopilot” 30
    31. 31. The Third Factor: Questions’ Proper Sequence Screener Ask for any final comments about any aspect of survey or Open-Ended Questions topic General Questions Ask for permission to follow- up with them about their Specific Questions answers Demographics Prompt if they have an issue they want to be contacted Follow-up about 31
    32. 32. The Third Factor: Questions’ Proper Sequence Screener Only at the very end ask respondent satisfaction Open-Ended Questions questions evaluating the survey General Questions Use to drive continual Specific Questions improvement to your Demographics research process itself Key measures: Follow-up How interesting was the survey How long was it Open comments 32
    33. 33. The Eightfold Path Right Attention: How Each Question Listens 33
    34. 34. Four Basic Question Types Open-Ended Questions Closed-Ended Questions Essay Question Fill in the Blank Choose One Choose Many 34
    35. 35. Open-Ended Questions vs. Closed-Ended Questions beige blue Bluw green purpel red fire engine Black blue red green purple red sea black blue Gray grey red green sky black blue Green orange red blue blue Blue r0x!! green orange red yellow 35
    36. 36. Best PracticesOpen-Ended Questions Closed-Ended Questions Great for hearing from the respondent in Make sure the list contains all common their own words choices Provides unbiased, unfiltered answers Better to have too many items rather than Good for catching anything you missed at too few, but try not to clutter the list end of a survey Provide the respondent with an “Other – Limit use, though, because: please specify” choice o Time consuming and taxing to Arrange the choices in logical order answer If no logical order, then o Difficult to analyze randomize the order 36
    37. 37. Yes/No Questions: Common Pitfalls Force-fitting a question into a yes/no format by overriding what “Yes” or “No” means. Asking for a single Yes/No to multiple items. Letting the user select both Yes and No. Providing caveats to the Yes/No choices. Asking questions that can’t be answered Yes/No. Listing a bunch of similar yes/no questions in a matrix. Asking questions that no one wants to say “no” to (“acquiescence response bias”). 37
    38. 38. Choose-Many Questions Use whenever more than one choice is applicable Always include a “none of the above” as an exclusive choice, otherwise can’t tell if respondent answered question Avoid providing choices that can be synonymous or subsets/supersets of one another Don’t use a list box to show the choices If the choice list has no natural order, randomize the order (anchoring “None of the above” to bottom) 38
    39. 39. Matrix/Table/Grid Questions Concise technique for combining questions with common topics Can be 50% faster for respondent to complete but speed may lead to mistakes Source: SSI, “Grid Test Summary”, 2009 39
    40. 40. Reimagine Matrix Questions when You Can"We know respondents dont like grids... Now, were beginning tolearn that not only are grids frustrating for respondents - they actuallyproduce inferior data." - Jackie Lorch, SSI VP Break each row of the matrix into a separate question or group of questions on its own page Rewrite each row of the matrix into separate questions, replacing checkboxes with fully labeled scales Refactor importance matrixes into choose-many questions Refactor Yes/No Matrix questions into choose-many questions (checkbox lists) instead Be creative 40
    41. 41. Mental Effort of Rating vs. Ranking QuestionsRanking Questions are More Taxing for Respondents 70 60 Rating Comparisons 50 Ranking 40 30 20 10 0 2 3 4 5 6 7 8 9 10 11 12 # Choices 41
    42. 42. When in Doubt, Use Rating QuestionsRanking Questions Rating Questions Taxing for respondents, requiring Lead to less differentiation among them to compare multiple items choices against one another Ratings often fall into narrow upper Difficulty increases disproportionately band as choices are added Personal variations in rating styles Take three times longer to answer Possible spurious positive correlations than rating questions (Munson and due to individuals’ personal variations McIntyre, 1979) Limit range of statistical analysis Alwin, D. F., & Krosnick, J. A. (1985). “The measurement of values in surveys: A comparison of ratings and rankings.” 42
    43. 43. Constant Sum & Allocation Questions Limit the number of items If you have too many items, break them into categories and then ask follow-on allocation questions If appropriate, include an Other category as a safety valve for respondents Use matrix questions instead if respondents cannot easily recollect and quantify their past behavior Provide visual feedback to the current working sum 43
    44. 44. The Eightfold Path Right Speech: Asking Objective Questions 44
    45. 45. The Fifth Factor: Right SpeechAsking Objective Questions Respondents should not be able to determine where you stand on any topic o Use nonjudgmental wording o Choose neutral terms Don’t ask leading questions o Not “What do you like about your service?” Nan-in served tea. He poured his visitors cup full, and then kept on pouring. The professor watched the overflow o But “What, if anything, do you like…?” until he no longer could restrain himself. "It is overfull. No more will go in!" Write from the respondent’s perspective not your "Like this cup," Nan-in said, "you are full of your own opinions and speculations. perspective How can I show you Zen unless you first empty your cup?" 45
    46. 46. The Fifth Factor: Right SpeechAsking Objective Questions Remove ambiguity: “What is your favorite drink?” (drink = beverage or drink = alcoholic beverage) Ask one item at a time not: “How would you rate our price and service?” not: “How easy to reach someone to help?” Avoid industry jargon Specify how you use general terms Don’t make subtle distinctions Have others proofread your questions for clarity Pre-test survey with a segment of your audience 46
    47. 47. The “Shoulds” of Question Wording Q. should be focused on a single issue or topic Q. should be interpreted the same way by all respondents Q. should use the respondent’s core vocabulary Q. should be a grammatically simple sentence if possible Q. should be brief 47
    48. 48. The “Should Nots” of Question Wording Q. should not assume criteria that are not obvious Q. should not be beyond the respondent’s ability or experience Q. should not use a specific example to represent a general case Q. should not request recall of specifics when only generalities will be remembered Q. should not require the respondent to guess a generalization Q. should not ask for details that cannot be related Q. should not contain words that overstate the condition Q. should not have ambiguous wording Q. should not be “double-barreled” Q. should not lead the respondent to a particular answer Q. should not have “loaded” wording or phrasing. 48
    49. 49. Test, Test, Test Self-Test Pre-test Pilot Test Publish Self-Test Question flow Answer validation Question wording Required answers Question types Skip patterns Scale consistency Errors of omission 49
    50. 50. The Eightfold Path Right Choice: Prompting with Care 50
    51. 51. What’s Your Favorite Color? Poll A – 6 Choices Poll H – 15 Choices Green Blue Purple Pink Green Orange Red Black White Blue Red Black Maroon Yellow Magenta Brown None of those! Gray Lime Green Sky Blue Hot Pink Whatever Hayleys favorite color is ;) 51
    52. 52. Provide Exhaustive Lists of Choices Top 3 Favorite Colors As Determined by Questions with Different Numbers of Choices 15 10 5 0 A B C D E F G H Correct Ranking 52
    53. 53. Source:Ocucom 53
    54. 54. Impact of Showing Logos May increase brand awareness recall May be more accurate for assessing print, web and video campaigns May be less accurate for assessing radio campaigns 54
    55. 55. To Label or Not Label Each Point of a ScaleMany Variations Possible Best Practices Respondents prefer fully labeled scales Fully labeled scales have greater reliability and validity Numeric values alter the meaning of labels and should be avoided 5-point unipolar and 7-point bipolar scales have greatest reliability and validity Where possible use standard scales rather than write your own Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” 55
    56. 56. Are Respondents Too Agreeable?Likert Scale Best Practices Completely disagree The traditional Likert scale is Disagree obsolete Somewhat disagree Over 100 studies have demonstrated acquiescence bias Neither agree or disagree Use “construct-specific response Somewhat agree options” instead – common rating Agree scales and custom scales Completely agree - Saris, Krosnick & Shaeffer (2005) 56
    57. 57. Patterns to Use for Scales “Hes embiggened that role with hisUnipolar Scale (0..100) cromulent Bipolar Scale (-1..0..+1) performance.” Completely* disembiggened Not at all cromulent Slightly cromulent Mostly disembiggened Somewhat disembiggened Moderately cromulent Neither embiggened nor Very cromulent disembiggened Completely* cromulent Somewhat embiggened Mostly embiggened*or “Extremely” where appropriate Completely* embiggened 57
    58. 58. Common Survey Rating Scales Acceptability Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable, Perfectly acceptable Agreement Completely disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Completely agree Amount of Use Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use Appropriateness Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate, Absolutely appropriate Awareness Not at all aware, Slightly aware, Moderately aware, Very aware, Extremely aware Beliefs Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat true of what I believe, True of what I believe, Very true of what I believe Concern Not at all concerned, Slightly concerned, Moderately concerned, Very concerned, Extremely concerned Familiarity Not at all familiar, Slightly familiar, Moderately familiar, Very familiar, Extremely familiar Frequency Never, Rarely, Sometimes, Often, Always Importance Not at all important, Slightly important, Moderately important, Very important, Extremely important Influence Not at all influential, Slightly influential, Moderately influential, Very influential, Extremely influential Likelihood Not at all likely, Slightly likely, Moderately likely, Very likely, Completely likely Priority Not a priority, Low priority, Medium priority, High priority, Essential Probability Not at all probable, Slightly probable, Moderately probable, Very probable, Completely probable Quality Very poor, Poor, Fair, Good, Excellent Reflect Me Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true of me Satisfaction (bipolar) Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat satisfied, Mostly satisfied, Completely satisfiedSatisfaction (unipolar) Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Completely satisfied 58
    59. 59. Developing Customer ScalesLabels Reflect Equal Intervals Common Scales100 Liking Very Poor, Poor, Fair, Good, Excellent 80 Amount 60 None, Some, Half, Most, All Frequency 40 Never, Rarely, Sometimes, Often, Always 20 Likelihood Not at all likely, Slightly likely, Moderately likely, 0 Very likely, Completely likely Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” 59
    60. 60. Use Dropdowns Only for Known Lists Great for letting respondents pick from a long list of known choices (e.g., states, provinces) Even casual users will use the keyboard effectively (e.g., click M three times to cycle from Maine to Maryland to Massachusetts) Make sure not to make any choice the default (e.g., having Alabama selected) Make the default choice an instruction like “Click here to choose” Don’t use when the choices have to be read to be understood, as in lists of industries or job titles 60
    61. 61. “Don’t Know” or “No Opinion” Choices When “satisficing”, respondents will select a no-opinion choice if presented if one, sometimes even if they have an opinion. When no such choice is presented, most respondents will choose from the other choices. Omit a no-opinion choice when asking for attitude. Provide a “Don’t Know” choice when prompting to recall specifics. Source: Krosnick, J. A., & Fabrigar, L. R. (1997). “Designing rating scales for effective measurement in surveys.” 61
    62. 62. Juxtaposing Next & Previous Buttons 1.20%Rate of Previous Button Use 1.00% 0.80% 0.60% 0.40% 0.20% 0.00% 9.4 9.5 9.6 9.7 9.8 9.9 10 10.1 10.2 10.3 Completion Time (minutes) Source: Couper, Baker, Mechling 62
    63. 63. Juxtaposing Next & Previous Buttons 1.20%Rate of Previous Button Use 1.00% 0.80% 0.60% 0.40% 0.20% 0.00% 0 2 4 6 8 10 12 Completion Time (minutes) Source: Couper, Baker, Mechling 63
    64. 64. The Eightfold Path Right Length: Not Too Short, Not Too Long 64
    65. 65. Right Length: Not Too Short, Not Too Long 2-4 questions: Transactional Survey 5-10 questions: Event Evaluation 10-20 questions: Customer Satisfaction 20-30 questions: Planning 50-70 questions: Major Account Review 70-90 questions: Employee Satisfaction 65
    66. 66. 66
    67. 67. The Sixth Habit: Shorten the Survey 100% Abandonment Rate Decrease the number 90% Of 180-Question of matrix/grid 80% Survey questions 70% 60% Reduce the number of 50% open-ended questions 40% Put demographic 30% questions at the end 20% Shorten the survey! 10% 0% Intro 10 12 13 14 15 16 17 18 19 20 1 2 3 4 5 6 7 8 9 11 Average 6 questions per block Source: “Dropouts on the Web”, Galesic, 2006 67
    68. 68. Causes of Survey Incompletes Primary Reason Respondent Abandoned Survey Subject matter 35% Media downloads 20% Survey length 20% Grids 15%Too many open-ends 5% 0% 20% 40% 60% 80% 100% Source: Lightspeed Research 68
    69. 69. Interesting Questionnaires are Perceived as Shorter Optimal 1 length 0.9 0.8 0.7 } perception gap response rates”, 2002, Mirta Galešic 0.6 “Effects of questionnaire length on Somewhat 0.5 too long Higher interest in questionnaire 0.4 Lower interest in questionnaire 0.3 0.2 Absolutely 0.1 Source: too long 0 Fewer Average More questions number of questions questions 69
    70. 70. Tips for Shortening the Survey Keep Your Focus – Remove questions that don’t directly address the goal of the survey Ask Only Most Important Questions – Common research tactic to have three similar questions on similar topic: use one Don’t Ask Esoteric Questions – Cut questions that make distinctions only apparent to those within your organization Don’t Set False Expectations – Remove questions that raise issues that can’t be addressed (for customers, free services; for employees, extended vacation time) 70
    71. 71. Shorten the Survey from the Respondent’s Perspective Skip Respondents Past Inapplicable Sections – Don’t subject respondent to survey about products or services they don’t have or can’t have Import Answers – Use CRM data to pipe in answers to “hidden questions” Randomize Displayed Sections – For less important sections, randomly display only one section to each respondent Break into Multiple Questionnaires – Maybe questions around different target groups are so different that they are best served with different questionnaires Use Fewer Pages – Page submits add a burden, so the fewer pages the better for the most part. Keep the Questionnaire Interesting – Respondents perceive interesting surveys as shorter! 71
    72. 72. The Eightfold Path Right Action: Sharing the Survey’s Karma 72
    73. 73. Right ActionSharing the Survey’s Karma Honor the time and energy of your respondents Share the results Change and grow to serve them better Use survey triggers to immediately take action 73
    74. 74. CRM Survey &System Panel Database Aggregate Responses Poor Rating Notification ManagerCustomer Survey with CRM Appended w/Customer Data Integration & Product Line Data or Customer Recovery 74
    75. 75. Use Survey Alerts/Email Triggers to Improve SatisfactionDon’t Just Measure Satisfaction—Intervene to Improve It! Email appropriate department when a respondent provides a negative rating of service Reactive o Help-desk ticket satisfaction o General dissatisfaction o Hotel-stay satisfaction o Major-account satisfaction Proactive o Customer-service satisfaction o Literature fulfillment 75
    76. 76. The Eightfold Path1. Right Perspective: The Respondent as Human2. Right Intention: Worthy Purpose for Survey3. Right Order: Questions in the Right Order4. Right Attention: How Each Question Listens5. Right Speech: Asking Objective Questions6. Right Choice: Prompting with Care7. Right Length: Not Too Short, Not Too Long8. Right Action: Fulfilling the Survey’s Karma 76
    77. 77. Questions & Answers To request a demonstration contact your sales executive or: 1-800-787-8755 sales@vovici.com I welcome your questions: Jeffrey Henning, jhenning@vovici.com Free e-book available from http://blog.vovici.com 77