Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Questionnaire Writing PROFESSIONAL DEVELOPMENT SERIES  - 2009
Note to users of this presentation. <ul><li>Market Researchers are freely welcome to use this presentation which I put tog...
Disclaimer <ul><li>While every care has been  taken, etc etc... Responsibility for any errors all mine...blah blah...usual...
Introduction <ul><li>There  is no element of market research so difficult as writing a questionnaire. </li></ul><ul><li>Yo...
Preparation for this Presentation <ul><li>Three main streams of input were combined to put this presentation together. </l...
<ul><li>Without proper theoretical training in research methods and statistics, and practical experience to know what does...
Personal Experience <ul><li>Involvement with phone surveys since 1986 </li></ul><ul><li>Face to face surveys since 1993 </...
Desk Research <ul><li>The Psychology of Survey Response . Roger Tourangeau, Lance J. Rips and Kenneth Rasinski. </li></ul>...
Colleague Input – online discussions <ul><li>Great input from the research colleagues on two  Linked in  groups. </li></ul...
BEFORE YOU EVEN START WRITING THE QUESTIONNAIRE
BEFORE YOU EVEN START WRITING THE QUESTIONNAIRE Think!
You need to think about all the stakeholders... NEEDS OF FIELD A clear roadmap that assists in communicating clearly. NEED...
Minimise Response Effects
Think about the client <ul><li>For the client we are designing a process that delivers measurement, guidance or inspiratio...
...to think about the analyst <ul><li>For the analyst we need to design a document that generates data of the highest qual...
...which is why you need to think about the field process <ul><li>For the field team (phone or online or face to face) we ...
... the respondent <ul><li>For the respondent we need to deliver an  experience  that respects where they’re at – and the ...
The questionnaire is the key to a whole process...
In the next section I’m going to discuss... <ul><li>The introduction of the questionnaire. </li></ul><ul><li>How to manage...
THE INTRODUCTION IS VITAL
Minimise Response Effects
Rapport to win co-operation. <ul><li>Respondent is judging us based on our preliminary contact. </li></ul><ul><ul><li>Are ...
The sense of contract in a questionnaire. <ul><li>I’ll give you the answers you require so long as: </li></ul><ul><ul><li>...
Putting people into the frame. <ul><ul><li>An introduction also serves to get people ramped into the subject matter. </li>...
Recommendations. <ul><li>Introduce yourself warmly using natural language. </li></ul><ul><li>Give your name (if its online...
MANAGE THE FLOW OF THE QUESTIONNAIRE EXPERIENCE
<ul><li>Before anything else, a questionnaire must first be a communication tool. If respondents fail to understand the qu...
Minimise Response Effects MANAGE THE FLOW OF THE QUESTIONNAIRE EXPERIENCE
Structure your survey carefully. <ul><li>Yes, keep it as short as possible – but short or long, develop a smooth logical f...
Verbal or visual signposts. <ul><ul><li>Clearly communicate the structure of the survey.  “This survey will ask about x, y...
Better quality, fewer drop outs. <ul><ul><li>Research is consistently showing that big arduous batteries of questions deli...
Recommendations. <ul><li>Introduce yourself warmly using natural language. </li></ul><ul><li>Give your name (if its online...
<ul><li>The farther we get away from paper questionnaires administered in a face-to-face methodology by an interviewer (us...
<ul><li>You definitely hit on a complete pet peeve that I forgot about! There is nothing more obvious or worse than a surv...
ASK THE RIGHT QUESTIONS
Minimise Response Effects ASK THE RIGHT QUESTIONS
The basics of a good question <ul><li>Questions need to be consistently understood. </li></ul><ul><li>Questions need to be...
<ul><li>KNOWLEDGE QUESTIONS  remembering memorizing recognizing recalling identification recalling information who, what, ...
Some rules of thumb about questions. <ul><li>For accuracy – make sure behavioural questions involve easily recalled timefr...
But the bigger problem is scales and code frames. <ul><li>What is the best way to code the respondents’ answers? </li></ul...
Some rules of thumb about scales. <ul><li>5 and 7 point scales are considered optimum* Labelling every point on the scale ...
<ul><li>A process of consciously going through a questionnaire and sense-checking everything. </li></ul>The poodle has 9 p...
Cognitive testing covers off 8 sets of potential problems – per question! <ul><li>Structure </li></ul><ul><li>How easy is ...
 
Guard against Irrelevancy. <ul><li>The survey feels “nothing to do with me.” </li></ul><ul><li>Make sure you’re surveying ...
Shorten the Length – real and perceived <ul><li>Surveys feel long if they zig-zag, feel repetitive, laborious or are...act...
Show some clear direction <ul><li>In today’s parlance, a questionnaire isn’t a questionnaire...it’s a journey. </li></ul><...
Laborious or forbidding questions <ul><li>Batteries, repetitive sets – cognitively hard questions. Too much can get respon...
Avoid Repetition <ul><li>There’s a fine line between having a rhythm to the questionnaire (you know the drill – now let’s ...
Slim it down. Avoid too much text <ul><li>However succinct you are – see if you can take more flab out. (My copy can usual...
Don’t treat respondents like they’re stupid. <ul><li>Any questionnaire that confuses simpleton for simplicity is pissing o...
 
The Cognitive process of answering a question Tourangeau and Rasinski have identified four basic cognitive steps in the wa...
The Cognitive process of answering a question Jon Krosnick has modified this outlook – suggesting that respondents might t...
Response order effects <ul><li>Primacy and recency – can effect the way we deal with and retrieve our thoughts.  </li></ul...
Acquiescence Bias <ul><li>A tendency to agree with unbalanced statements. </li></ul><ul><li>The satisficing theory suggest...
Ranking tests <ul><li>Ranking exercises impose a significant cognitive burden – and are not very popular. </li></ul><ul><l...
The presence of a “No Opinion” filter. <ul><li>Traditionally we’re trained to include “No Opinion” as an option. </li></ul...
Timeframe – the burden of memory recall <ul><li>People have faulty memories. </li></ul><ul><li>If the subject is not recen...
<ul><li>Don't ask respondents to recall specifics about their shopping behaviour more than a week or two in the past. For ...
Cognitive booster 1. Maximising Respondent Motivation
Cognitive booster 2. Minimise task difficulty
Cognitive booster 3. Minimise Response Effects
 
Avoid huge batteries. <ul><li>Tiresome to fill in. </li></ul><ul><li>Encourages satisficing – just tick the boxes. </li></...
<ul><ul><li>Thinking back to your experience with Shop X, please indicate your degree of agreement with the following stat...
NONE OF THESE BRANDS THIS DRAG AND DROP EXERCISE IS MORE INTUITIVE – AND ACTUALLY HANDLES MORE DATA THAN THE PREVIOUS PAGE...
 
Open enders <ul><li>The best listening tool there is. </li></ul><ul><li>Captures opinions and ideas outside of our initial...
Don’t ask them if you’re not going to use them. <ul><li>People LIKE filling in the occasion open-ender – but don’t like th...
 
<ul><li>The best project folk I know are those that started out in coding departments (which don't really exist anymore) o...
What is analysis? <ul><li>Basically analysis is the process of converting raw numbers or words back into the format they e...
Talk with them about the outputs for the client. <ul><li>What sort of data will they be able to analyse?  </li></ul><ul><l...
Some questions create more work. Others less. <ul><li>Single choice questions are way easier to work with than multichoice...
Advanced analytics have special requirements. <ul><li>Factor analysis requires biggish batteries of questions. Or is that ...
Then there’s the basics... <ul><li>Keep all your scales running in the same direction. (Big = good is most intuitive for m...
NEARLY THERE.... Before you go into field!
<ul><li>Develop and review the questionnaire as a separate document. I've seen too many people trying to program before th...
Test and test again. <ul><ul><li>Allow time for adequate testing. </li></ul></ul><ul><ul><li>Check that the logic works. <...
<ul><li>Prepare a detailed outline of how you want the report to look, this is like a great insurance policy against the m...
Keep learning!
Expose yourself to feedback. <ul><li>For a listening profession, we don’t always listen. </li></ul><ul><li>At the end of q...
Expose yourself to feedback. <ul><li>“ Great survey, it is not until you do one of these you realise how much money you sp...
Keep up your personal research. <ul><li>There are plenty of books on the subject. </li></ul><ul><li>There is limitless mat...
YOU THOUGHT WE’D NEVER GET HERE! Summary!
Summary <ul><li>Questionnaires are the single most difficult and riskiest task of the researcher.  </li></ul><ul><li>Quest...
<ul><li>Construct questionnaire from general to specific - e.g. overall brand/concept ratings following by specific rating...
Duncan Stuart  FMRSNZ <ul><li>2004 - elected as a Fellow of the MRSNZ. </li></ul><ul><li>Twice judge, NZ marketing Awards....
Thank you! Duncan Stuart  FMRSNZ [email_address] Telephone 64 9 366 0620 www.kudos-dynamics.com If you find this presentat...
Upcoming SlideShare
Loading in …5
×

Questionnaire Writing Workshop 97 Version

1,803 views

Published on

  • Duncan Stuart of Kudos Organisational Dynamics Ltd in New Zealand put together the following presentation based on feedback from the Next Gen Market Research Group (NGMR) on LinkedIn. Thanks for the initiative Duncan!

    To find about membership in NGMR please see http://www.linkedin.com/e/gis/31804

    Tom
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • To find about membership in NGMR please see http://www.linkedin.com/e/gis/31804
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Duncan Stuart of Kudos Organisational Dynamics Ltd in New Zealand put together the following presentation based on feedback from the Next Gen Market Research Group (NGMR) on LinkedIn. Thanks for the initiative Duncan!
    Tom
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Questionnaire Writing Workshop 97 Version

  1. 1. Questionnaire Writing PROFESSIONAL DEVELOPMENT SERIES - 2009
  2. 2. Note to users of this presentation. <ul><li>Market Researchers are freely welcome to use this presentation which I put together for a professional Development session in New Zealand in April 2009. </li></ul><ul><li>It is a long presentation – so you might be wise to trim it to suit your needs. </li></ul><ul><li>Please credit those who have been quoted, and an acknowledgement to the MRSNZ would be suitable too as well as to Tom H. C. Anderson who hosts the Linked In group Next Generation Market Researchers: whose members kindly contributed much thought and discussion to this presentation. </li></ul><ul><li>Otherwise go for it! We ought to share more notes professionally. </li></ul><ul><li>One other request. If you find this presentation useful, and it saves you valuable time – then please consider making a donation to my cause – a school I sponsor in Cambodia. Savong’s School is a great project www.savong.com </li></ul><ul><li>Duncan Stuart FMRSNZ </li></ul>
  3. 3. Disclaimer <ul><li>While every care has been taken, etc etc... Responsibility for any errors all mine...blah blah...usual caution... Please use with care, failure to do so...result in sudden death ...sacking from employer etc etc... No hurting of children or animals .... Unintentional damage to property... Public humiliation... Will try not to run over-time in this presentation... No liability...good luck etc....better check with the lawyers </li></ul>Duncan Stuart accepts full blame for any errors, omissions or oversights in this presentation. If you have any suggestions or questions please feel free to contact me. [email_address]
  4. 4. Introduction <ul><li>There is no element of market research so difficult as writing a questionnaire. </li></ul><ul><li>You face a blank sheet of paper and you are required to “architect” a full experience for the respondent, a solid platform for the analyst, and do these things in a way that will deliver the solutions needed by the client. Underneath it all, gnawing away at the exercise, are two ghastly truths. </li></ul><ul><ul><li>The way you write your questionnaire is doomed to be a compromise of length, style, demands and human biases. </li></ul></ul><ul><ul><li>GIGO. Garbage in. Garbage out. If you write an inferior questionnaire the whole project will deliver bad value. </li></ul></ul><ul><li>Good luck! </li></ul>PROFESSIONAL DEVELOPMENT SERIES - 2009
  5. 5. Preparation for this Presentation <ul><li>Three main streams of input were combined to put this presentation together. </li></ul>
  6. 6. <ul><li>Without proper theoretical training in research methods and statistics, and practical experience to know what does and doesn't work in practice, I don't know how anybody can really conduct sound research except by chance...which I believe we see a lot of in the market research industry </li></ul>Peter Hill Vice President of Research & Analysis “
  7. 7. Personal Experience <ul><li>Involvement with phone surveys since 1986 </li></ul><ul><li>Face to face surveys since 1993 </li></ul><ul><li>Online surveys since 1998 </li></ul>
  8. 8. Desk Research <ul><li>The Psychology of Survey Response . Roger Tourangeau, Lance J. Rips and Kenneth Rasinski. </li></ul><ul><li>Improving Survey Questions . Floyd J. Fowler. </li></ul><ul><li>The threat of Satisficing in Surveys : The Shortcuts Respondents take in Answering Questions. Jon Krosnick. </li></ul><ul><li>Cognitive Interviewing . Gordon B.Willis. </li></ul>
  9. 9. Colleague Input – online discussions <ul><li>Great input from the research colleagues on two Linked in groups. </li></ul><ul><li>Next Gen Market Research (NGMR) </li></ul><ul><li>Market Research Knowledge Forum. </li></ul>
  10. 10. BEFORE YOU EVEN START WRITING THE QUESTIONNAIRE
  11. 11. BEFORE YOU EVEN START WRITING THE QUESTIONNAIRE Think!
  12. 12. You need to think about all the stakeholders... NEEDS OF FIELD A clear roadmap that assists in communicating clearly. NEEDS OF CLIENTS I want answers to my problem NEEDS OF ANALYSTS I need data that is structured in a useful way to test hypotheses. NEEDS OF RESPONDENTS Please make it short, relevant , clear and give me room to share my true opinion. YOU ARE HERE
  13. 13. Minimise Response Effects
  14. 14. Think about the client <ul><li>For the client we are designing a process that delivers measurement, guidance or inspiration – sometimes all three. </li></ul><ul><ul><li>Are they after results that measure something? </li></ul></ul><ul><ul><li>Or results that guide them on a decision? </li></ul></ul><ul><ul><li>Or is this an idea generation process? </li></ul></ul><ul><li>Be very clear about what you’re trying to achieve. Then work backwards from there... </li></ul>
  15. 15. ...to think about the analyst <ul><li>For the analyst we need to design a document that generates data of the highest quality through questions that are relevant meaningful and answer-options that are crunch-able in order to deliver the measurement, guidance or inspiration required by the client. </li></ul><ul><li>What will the analysis look like? What kinds of slides will you be generating? What format do they need the data in? </li></ul><ul><li>Will they have a lot of time to work with the data – or just a short timeframe? (You can simplify things for them if necessary.) </li></ul><ul><li>And part of that comes back a step... </li></ul>
  16. 16. ...which is why you need to think about the field process <ul><li>For the field team (phone or online or face to face) we need to deliver a document that acts as a well designed roadmap that allows us to contact the right people and then doesn’t lead anyone astray or into grief. </li></ul><ul><li>Who are we talking to? </li></ul><ul><li>How do we filter out those we don’t want? </li></ul><ul><li>What limitations does the field process put on the questionnaire design? (Length? Language? Ability to show pictures?) </li></ul><ul><li>And of course we need also to think about... </li></ul>
  17. 17. ... the respondent <ul><li>For the respondent we need to deliver an experience that respects where they’re at – and the fact that they’re neither stupid or tolerant of being bored. </li></ul><ul><li>We need to communicate (instruct, ask and listen) in a manner that is as clear and unambiguous as possible. </li></ul>
  18. 18. The questionnaire is the key to a whole process...
  19. 19. In the next section I’m going to discuss... <ul><li>The introduction of the questionnaire. </li></ul><ul><li>How to manage the flow of the questionnaire experience. </li></ul><ul><li>Question writing – the art and the science, the good and the bad (scales, question styles...) </li></ul><ul><li>Respondent fatigue and how to minimise it. </li></ul><ul><li>Respondent biases – how to reduce these. </li></ul>
  20. 20. THE INTRODUCTION IS VITAL
  21. 21. Minimise Response Effects
  22. 22. Rapport to win co-operation. <ul><li>Respondent is judging us based on our preliminary contact. </li></ul><ul><ul><li>Are we relevant? </li></ul></ul><ul><ul><li>Are we approachable? (Will we bite?) </li></ul></ul><ul><ul><li>Will this be an engaging experience? </li></ul></ul><ul><ul><li>Should I agree to do this? </li></ul></ul>
  23. 23. The sense of contract in a questionnaire. <ul><li>I’ll give you the answers you require so long as: </li></ul><ul><ul><li>You don’t start probing into issues that I find too deeply personal. </li></ul></ul><ul><ul><li>You don’t take undue time or cause me undue inconvenience. </li></ul></ul><ul><ul><li>You promise to report my feelings fairly – and that also means you’d better ask fair questions that don’t paint me into a box. </li></ul></ul><ul><ul><li>You attempt to make the exercise even a bit entertaining or engaging. </li></ul></ul><ul><ul><li>You promise I will not be harmed, bothered or put on some damned list as a result of taking part in this exercise. </li></ul></ul>
  24. 24. Putting people into the frame. <ul><ul><li>An introduction also serves to get people ramped into the subject matter. </li></ul></ul><ul><ul><li>This is about opening up their memory-bank and giving them the opportunity to “tune-in” to the topic. </li></ul></ul><ul><ul><li>A significant part of whole survey design is about this cognitive “tuning in process” –and it begins right here. </li></ul></ul>
  25. 25. Recommendations. <ul><li>Introduce yourself warmly using natural language. </li></ul><ul><li>Give your name (if its online, add your photo, make it a personal contact.) Dial up the rapport! Give a little of yourself. </li></ul><ul><li>Share the purpose of the survey. “I have a client who makes peanut butter – and he wants to find out x, y and z. I hope you can help me.” </li></ul><ul><li>Cover off the elements that are in the implied contract. (Privacy, confidentiality.) </li></ul><ul><li>Hint at some of the subjects you’ll raise, and what to expect in the survey. “First I’ll ask what you currently eat – and then later, I’d like to show you some new pack designs...” </li></ul>Online surveys are inherently more personal than CATI or even Face to Face. They communicate with a single voice – your own, while other surveys are conducted by multiple voices that need to be “scripted neutral.”
  26. 26. MANAGE THE FLOW OF THE QUESTIONNAIRE EXPERIENCE
  27. 27. <ul><li>Before anything else, a questionnaire must first be a communication tool. If respondents fail to understand the questions as intended, everything else falls apart. I work with a lot of suppliers and see a lot of questionnaires, and this is clearly one of the greatest weaknesses in the world of research today. Are we so focused on the back-end analytics that we forget about this? </li></ul>Dan Womack Senior Manager, Insights; AFLAC Worldwide HQ “
  28. 28. Minimise Response Effects MANAGE THE FLOW OF THE QUESTIONNAIRE EXPERIENCE
  29. 29. Structure your survey carefully. <ul><li>Yes, keep it as short as possible – but short or long, develop a smooth logical flow. </li></ul><ul><li>Don’t zigzag or hopscotch. Try to be linear – for example start macro and slowly zoom in to the heart of the subject. E.g. start with the category, then the brand, then the specifics of the brand. </li></ul><ul><ul><li>This is smoother and also helps cognitive flow. </li></ul></ul><ul><li>If any sections are a bit arduous, create a change of activity – so there’s rhythm in the experience. </li></ul><ul><ul><li>For example if there’s a somewhat repetitive task in the questionnaire, provide a “breather activity” that is easier and lighter and helps you gear-shift into the next activity. </li></ul></ul>
  30. 30. Verbal or visual signposts. <ul><ul><li>Clearly communicate the structure of the survey. “This survey will ask about x, y, and z and then we’ll wrap up by asking your opinion of some new pack designs.” </li></ul></ul><ul><ul><li>Let respondents know that they’re making good progress and that the end is in sight. Progress bars and verbal cues help. </li></ul></ul><ul><ul><li>Set localised expectations. “ Next I’m going to ask you 6 questions about x – and then we’ll look at those new pack designs I want to show you.” </li></ul></ul><ul><ul><li>Provide encouragement. “Thanks for that – in fact we’re 80% of the way through the survey now.” </li></ul></ul>My own signposts tend to be a bit wordy – and respondents tell me that! That doesn’t mean exclude signposts – but keep them succinct...
  31. 31. Better quality, fewer drop outs. <ul><ul><li>Research is consistently showing that big arduous batteries of questions deliver dulled data quality. They encourage respondents to offer less discriminant answers. </li></ul></ul><ul><ul><li>Surveys that “feel” too long result in respondent drop-out: which is a waste of everybody’s time. </li></ul></ul><ul><ul><li>Surveys that get repetitive, or too zig-zaggy or simply too arduous all trigger the same warning light to the respondent: “Warning – no light, ever, at the end of this tunnel.” </li></ul></ul>Big batteries are too easy to write! They seduce us into covering off the brief with big long lists. They consistently produce inferior data.
  32. 32. Recommendations. <ul><li>Introduce yourself warmly using natural language. </li></ul><ul><li>Give your name (if its online, add your photo, make it a personal contact.) </li></ul><ul><li>Share the purpose of the survey. “I have a client who makes peanut butter – and he wants to find out x, y and z. Are you able to help me?” </li></ul><ul><li>Cover off the elements that are in the implied contract. </li></ul><ul><li>Hint at some of the issues you’ll raise, and what to expect in the survey. “Later, I’d like to show you some new pack designs...” </li></ul>Online surveys are inherently more personal than CATI or even Face to Face. They communicate with a single voice – your own, while other surveys are conducted by multiple voices that need to be “scripted neutral.”
  33. 33. <ul><li>The farther we get away from paper questionnaires administered in a face-to-face methodology by an interviewer (usually a housewife) using a clipboard and pencil, which are then manually edited and coded, perhaps on a kitchen table -- and the closer we get to programmed self-administered questionnaires that are never actually touched or seen when completed -- the more likely it is that we have questionnaire writers who don't know how to write an effective and engaging (and &quot;respectful&quot;) questionnaire. </li></ul>Dan Gersten Owner, Dan Gersten & Associates, LLC “
  34. 34. <ul><li>You definitely hit on a complete pet peeve that I forgot about! There is nothing more obvious or worse than a survey that has been defined for one mode and administered in another. There are significant question and answer/scale construction differences for telephone, on-line, face-to-face and self-administered. It really bugs me when I get a survey intended for the internet delivered by phone. Some survey software provides &quot;seamless administration of surveys across multiple modes&quot;...but this definitely doesn't mean that the same survey should be used. </li></ul>Peter Hill Vice President of Research & Analysis “
  35. 35. ASK THE RIGHT QUESTIONS
  36. 36. Minimise Response Effects ASK THE RIGHT QUESTIONS
  37. 37. The basics of a good question <ul><li>Questions need to be consistently understood. </li></ul><ul><li>Questions need to be administered or communicated in a consistent fashion. </li></ul><ul><li>What it is that constitutes an adequate answer needs to be consistently communicated. </li></ul><ul><li>All respondents should have access to the information needed to answer the question. (Unless we’re actually trying to measure level of knowledge.) </li></ul><ul><li>Respondents must be willing to provide the answers called for in the question. </li></ul><ul><li>It must be reasonably engaging. (See over...) </li></ul>
  38. 38. <ul><li>KNOWLEDGE QUESTIONS remembering memorizing recognizing recalling identification recalling information who, what, when, where, how ...? describe </li></ul><ul><li>COMPREHENSION QUESTIONS interpreting translating from one medium to another describing in one's own words organisation and selection of facts and ideas retell... </li></ul><ul><li>APPLICATION QUESTIONS problem solving applying information to produce some result use of facts, rules and principles how is ... an example of ...? how is ... related to ...? why is ... significant? </li></ul><ul><li>ANALYSIS QUESTIONS subdividing something to show how it is put together finding the underlying structure of a communication identifying motives separation of a whole into component parts what are the parts or features of ...? classify ... according to ... outline/diagram ... how does ... compare/contrast with ...? what evidence can you list for ...? </li></ul><ul><li>SYNTHESIS QUESTIONS creating a unique, original product that may be in verbal form or may be a physical object combination of ideas to form a new whole what would you predict/infer from ...? what ideas can you add to ...? how would you create/design a new ...? what might happen if you combined ...? what solutions would you suggest for ...? </li></ul><ul><li>EVALUATION QUESTIONS making value decisions, opinions, judgements or decisions. Do you agree that ...? what do you think about ...? what is the most important ...? place the following in order of priority ... how would you decide about ...? what criteria would you use to assess ...? </li></ul>BEHAVIOURAL QUESTIONS IDEA GENERATION QUESTIONS EVALUATION QUESTIONS ATTITUDINAL QUESTION EVALUATION QUESTION DEEP PROBE - INTERPRETATION DEEP PROBE - INTERPRETATION
  39. 39. Some rules of thumb about questions. <ul><li>For accuracy – make sure behavioural questions involve easily recalled timeframes. </li></ul><ul><ul><li>Last week is better than “last month” For that matter “In a typical week” is more accurate than “last week.” </li></ul></ul><ul><ul><li>Focus on the immediate past for behaviour – predictive questions are notoriously unreliable. They’re an indicator at best. </li></ul></ul><ul><li>Avoid double barrelled questions. </li></ul><ul><li>Avoid value judgements or implied judgements. </li></ul><ul><li>Provide alternatives. Do you support stricter gun control – or less strict gun control? </li></ul>How many litres of milk have you purchased in the past decade? Do you feel the church and politicians should speak out against tobacco and crime? Do you favour higher taxes?
  40. 40. But the bigger problem is scales and code frames. <ul><li>What is the best way to code the respondents’ answers? </li></ul><ul><li>When do I use a five point scale? </li></ul><ul><li>Should I label every point? </li></ul><ul><li>Is a bi-polar scale worse or better than a uni-polar scale (Don’t agree to fully agree?) </li></ul><ul><li>Should I use rating – or ranking? </li></ul>
  41. 41. Some rules of thumb about scales. <ul><li>5 and 7 point scales are considered optimum* Labelling every point on the scale (5pt – 7pt) is desirable. </li></ul><ul><li>10 point scales are good intuitive ways of rating something overall “out of 10...” because it is part of our vernacular. </li></ul><ul><li>10 point scales are better when they start at 0 and are actually 11 point scales. (They have a mid way point.) </li></ul><ul><li>100 point scales offer little if any useful advantage – and are more complex to analyse. (They probably need recoding.) </li></ul><ul><li>Having a neutral or half way point is strongly desirable because many people feel this way. </li></ul><ul><li>A three point scale (Agree/Neither-Nor/Disagree) is almost always way too blunt and not discriminant enough. </li></ul><ul><li>*There’s debate about this still. In Europe 11 pt scales are much more widely used. </li></ul><ul><li>Typical Likert Scale </li></ul><ul><ul><li>Strongly disagree </li></ul></ul><ul><ul><li>Disagree </li></ul></ul><ul><ul><li>Neither agree or disagree </li></ul></ul><ul><ul><li>Agree </li></ul></ul><ul><ul><li>Strongly Agree </li></ul></ul>Where did the Likert scale come from? Invented by Rensis Likert (1903–1981). In 1932 he received his Ph.D in psychology from Columbia University. For his thesis work, Likert produced a survey scale ( Likert Scales ) as a means of measuring attitudes, showing that it captured more information than competing methods. (Wikipedia)
  42. 42. <ul><li>A process of consciously going through a questionnaire and sense-checking everything. </li></ul>The poodle has 9 puppies. The collie dog has 5 puppies. How many more puppies does the poodle have? Answer. None I guess. Perhaps she stopped at 9.
  43. 43. Cognitive testing covers off 8 sets of potential problems – per question! <ul><li>Structure </li></ul><ul><li>How easy is the question to read? </li></ul><ul><li>What assumed information is missing? </li></ul><ul><li>Content </li></ul><ul><li>Is the intent of the question clear? </li></ul><ul><li>Is the word precise and clear – or mushy and ungrammatical? </li></ul><ul><li>Is the question unambiguous. </li></ul><ul><li>If time is an element (“how often have you recently...”) is the reference point clear. How recent is recent? </li></ul><ul><li>Assumptions. </li></ul><ul><li>Does the question assume anything about the respondent? </li></ul><ul><li>Double-barrelling: is more than one question explicit or implied? </li></ul><ul><li>Instructions </li></ul><ul><li>Are the instructions complicated, unclear, or conflicting? </li></ul><ul><li>Knowledge/Memory </li></ul><ul><li>Are we assuming respondent has attitude or knowledge of subject? </li></ul><ul><li>Are we assuming respondent can recall details or information? </li></ul><ul><li>Are we assuming they can compute the answer? </li></ul><ul><li>Sensitivity/Bias </li></ul><ul><li>Embarrassing topics </li></ul><ul><li>Social Acceptability issues </li></ul><ul><li>Response Categories </li></ul><ul><li>Difficult open enders? </li></ul><ul><li>Overlapping categories? </li></ul><ul><li>Missing categories? </li></ul><ul><li>Too many or hard to decipher categories? </li></ul><ul><li>Other problems </li></ul><ul><li>Ordering problems </li></ul><ul><li>Technical issues </li></ul>
  44. 45. Guard against Irrelevancy. <ul><li>The survey feels “nothing to do with me.” </li></ul><ul><li>Make sure you’re surveying the right people. </li></ul><ul><li>Pique their interest about the topic. Paper towels or toilet cleaners or politics – it can all be engaging. </li></ul>
  45. 46. Shorten the Length – real and perceived <ul><li>Surveys feel long if they zig-zag, feel repetitive, laborious or are...actually long! </li></ul><ul><li>A long experience is more tolerable if it is engaging and even entertaining. </li></ul><ul><li>Vary the activities, vary the complexity and give respondents a sense of direction. (See over) </li></ul>
  46. 47. Show some clear direction <ul><li>In today’s parlance, a questionnaire isn’t a questionnaire...it’s a journey. </li></ul><ul><li>Give it a clear beginning, middle, mountain to climb...and then ...an ending. </li></ul>
  47. 48. Laborious or forbidding questions <ul><li>Batteries, repetitive sets – cognitively hard questions. Too much can get respondents to quit. </li></ul><ul><li>Intersperse harder questions with easier questions. </li></ul><ul><li>Break down hard issues into smaller chunks. </li></ul><ul><li>Explain what you’re doing. “I’d like to ask you this from another angle – imagine you were serving wine and a wine snob was one of your guests...now what would you serve?” </li></ul>
  48. 49. Avoid Repetition <ul><li>There’s a fine line between having a rhythm to the questionnaire (you know the drill – now let’s do it again for Brand Y) versus tedious repetition. </li></ul>
  49. 50. Slim it down. Avoid too much text <ul><li>However succinct you are – see if you can take more flab out. (My copy can usually lose 25-30%). Do a word count on the questionnaire – be assiduous about paring out unnecessary verbiage. Shorter, simpler copy is easier for everyone. </li></ul>
  50. 51. Don’t treat respondents like they’re stupid. <ul><li>Any questionnaire that confuses simpleton for simplicity is pissing off the respondent – big time. </li></ul><ul><li>Caused by inadequate answer options, forced choices. sing song La la La “Low IQ” links and stupid questions – do you think we should have better government? </li></ul><ul><li>Treat symptoms with intelligence. Confide with the respondent – talk to them off the record between questions – let them know what issues the client is thinking about. (They want to test a new pack to help lift sales – but they need your opinion...) </li></ul><ul><li>Avoid jargon and marketing speak. </li></ul>
  51. 53. The Cognitive process of answering a question Tourangeau and Rasinski have identified four basic cognitive steps in the way we process questions.
  52. 54. The Cognitive process of answering a question Jon Krosnick has modified this outlook – suggesting that respondents might tend to “satisfice” – that is, give a ‘good enough’ answer by shortcutting the process. EFFICIENT RESPONSE The “satisficing instinct” can greatly reduce the precision of the data. We need to systematically address this to lift survey performance. Get the gist. Save effort.
  53. 55. Response order effects <ul><li>Primacy and recency – can effect the way we deal with and retrieve our thoughts. </li></ul><ul><li>Primacy effects occur more in written surveys. First mentioned gets more votes. </li></ul><ul><li>Recency effects occur more often in phone surveys. Last mentioned gets more votes. </li></ul>
  54. 56. Acquiescence Bias <ul><li>A tendency to agree with unbalanced statements. </li></ul><ul><li>The satisficing theory suggests that respondents tend to assume that the main option on offer is probably the “correct” option. The easiest response is simply to “agree.” </li></ul>
  55. 57. Ranking tests <ul><li>Ranking exercises impose a significant cognitive burden – and are not very popular. </li></ul><ul><li>The results tend to be more accurate however – compared to batteries of rating scales. </li></ul><ul><li>The answer is to use them sparingly – and keep them relatively simple. (5 or 6 items is okay.) </li></ul>
  56. 58. The presence of a “No Opinion” filter. <ul><li>Traditionally we’re trained to include “No Opinion” as an option. </li></ul><ul><li>Findings suggest that most people who offer “no opinion” do have an opinion. By removing the option you encourage a more thoughtful answer. </li></ul>
  57. 59. Timeframe – the burden of memory recall <ul><li>People have faulty memories. </li></ul><ul><li>If the subject is not recent then memories can have an unreliable telescoping effect. </li></ul><ul><li>The cognitive burden can be reduced through using reference points (since Christmas...) and landmarks. </li></ul><ul><li>Other ways to reduce burden are branching formats (you say you played more last year...was that before or after September?) as well as simple probes: (Any other?) </li></ul>
  58. 60. <ul><li>Don't ask respondents to recall specifics about their shopping behaviour more than a week or two in the past. For example, how many of your purchases in the past year were made with a coupon? It's unrealistic to expect anyone to remember that. I can barely remember what coupons I used on my shopping trip earlier today. Similarly, asking shopping questions like &quot;how did you chose that brand of toothbrush?&quot; ... unless someone is in the store in front of the shelf, they won't correctly recall why they chose that brand out of a hundred options on the shelf. </li></ul>Valerie Skala Walker Marketing / Product Management / Market Research “
  59. 61. Cognitive booster 1. Maximising Respondent Motivation
  60. 62. Cognitive booster 2. Minimise task difficulty
  61. 63. Cognitive booster 3. Minimise Response Effects
  62. 65. Avoid huge batteries. <ul><li>Tiresome to fill in. </li></ul><ul><li>Encourages satisficing – just tick the boxes. </li></ul><ul><li>Anchoring effect: your answer for the first question influences your answers for the rest of the battery. </li></ul>
  63. 66. <ul><ul><li>Thinking back to your experience with Shop X, please indicate your degree of agreement with the following statements. </li></ul></ul>Strongly agree Agree Neutral Somewhat disagree Disagree Strongly disagree Not sure <ul><ul><li>I prefer making a purchase from Shop X to using local offices, malls or stores. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I prefer Shop X over other home shopping services (i.e., catalogs, &quot;1-800&quot; services or television). </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>Shop X doesn't just sell products or services--it entertains me. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I received special rewards and discounts from doing business with Shop X </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I say positive things about Shop X to other people. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I consider Shop X to be my first choice when I need products or services of this type. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>The &quot;look&quot; of the Shop X web site is appealing to me. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I really like doing business with Shop X </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I intend to continue to visit the Shop X site in the future. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>I intend to purchase from Shop X in the future. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>Shop X is one of the first places I intend to look when I need the type of merchandise or services it provides. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>It would require a lot of time and effort on my part, to set up an account with another shop . </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>It would take a lot of time and energy to look for another retailer for this type of product. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο <ul><ul><li>The products and/or services I purchased from Shop X were a good value. </li></ul></ul>Ο Ο Ο Ο Ο Ο Ο
  64. 67. NONE OF THESE BRANDS THIS DRAG AND DROP EXERCISE IS MORE INTUITIVE – AND ACTUALLY HANDLES MORE DATA THAN THE PREVIOUS PAGE. Here’s a sorting exercise: which words go with which brand? Simply drag the words into the appropriate bucket below.
  65. 69. Open enders <ul><li>The best listening tool there is. </li></ul><ul><li>Captures opinions and ideas outside of our initial scope. </li></ul><ul><li>Gives the human nuance to the data (helps contextualise the numbers.) </li></ul><ul><li>Ideal for opinion, decision-focused and idea generation research. </li></ul>
  66. 70. Don’t ask them if you’re not going to use them. <ul><li>People LIKE filling in the occasion open-ender – but don’t like the burden of too many. </li></ul><ul><li>Don’t make these compulsory. If they’ve got nothing to say – don’t force them to say it. </li></ul><ul><li>Usually (not always) these questions are for colouration – not numbers. </li></ul>
  67. 72. <ul><li>The best project folk I know are those that started out in coding departments (which don't really exist anymore) or field departments or in some other area like graphics. Those are the basics, that all too often were never learned in order to have been forgotten. IMO, it is extremely helpful and valuable for a person who is writing a questionnaire to have an understanding of how the questionnaire is going to be administered, how the data is to be collected, how the data is to be processed and managed and analyzed and presented. </li></ul>Dan Gersten Owner, Dan Gersten & Associates, LLC “
  68. 73. What is analysis? <ul><li>Basically analysis is the process of converting raw numbers or words back into the format they existed in when the respondents were asked the questions. </li></ul><ul><li>We ask people to express their feelings “out of 10” – and the analyst’s job is to translate those collective numbers back into feelings – and to make sense of what they’re hearing. </li></ul>
  69. 74. Talk with them about the outputs for the client. <ul><li>What sort of data will they be able to analyse? </li></ul><ul><li>Is it </li></ul><ul><ul><li>about measurement, </li></ul></ul><ul><ul><li>about decision-making or </li></ul></ul><ul><ul><li>about idea generation? </li></ul></ul><ul><li>Will you have an adequate timeframe for analysis? </li></ul>
  70. 75. Some questions create more work. Others less. <ul><li>Single choice questions are way easier to work with than multichoice (tick any that apply) questions. </li></ul><ul><li>Big wide scales (out of 10 or 100) probably need recoding into a chunkier scale. </li></ul><ul><li>Open enders require coding. </li></ul><ul><li>Complex branches or skips can create patchy spreadsheets with ever-changing bases. Much harder to analyse. </li></ul>
  71. 76. Advanced analytics have special requirements. <ul><li>Factor analysis requires biggish batteries of questions. Or is that vice versa? </li></ul><ul><li>Perceptual maps usually require at least 5 or 6 variables asked on the same scale. </li></ul><ul><li>Segmentations can require similar batteries – 5 or six variables for example. </li></ul>
  72. 77. Then there’s the basics... <ul><li>Keep all your scales running in the same direction. (Big = good is most intuitive for most people.) </li></ul><ul><li>Try and standardise your scales to an appropriate extent. </li></ul>
  73. 78. NEARLY THERE.... Before you go into field!
  74. 79. <ul><li>Develop and review the questionnaire as a separate document. I've seen too many people trying to program before the questionnaire is nailed down. That's inefficient and likely to lead to a survey that doesn't hit the spot. </li></ul><ul><li>Don't ask for Day/Month/Year of birth. I haven't yet found any valid reason why researchers would do this. It just seems like laziness, and it raises concerns about person information that will reduce response rates. It is generally best to offer a range of ages. If you absolutely must ask for the year (because you don't know how to cut the results?) just stick to the year. </li></ul><ul><li>Do your very best to make sure that options are realistic for the respondent. This probably means putting in enough effort into pre-testing among outsiders, and possibly adding Other/Specifies. I see far too many surveys - and distressingly even from professionals - where the results are compromised because the respondent is forced to choose an option that isn't accurate. Overall, to paraphrase others' comments - be respectful of the respondent, and think of the survey as a conversation. </li></ul>Mike Pritchard Experienced Insightful Market Researcher - Principal at 5 Circles Research “
  75. 80. Test and test again. <ul><ul><li>Allow time for adequate testing. </li></ul></ul><ul><ul><li>Check that the logic works. </li></ul></ul><ul><ul><li>Check that your spelling is correct. </li></ul></ul><ul><ul><li>Check that you didn’t cut and paste the wrong scales to the wrong questions. </li></ul></ul><ul><ul><li>Check that non-marketers understand the questions. </li></ul></ul><ul><ul><li>Conduct cognitive interviews to thoroughly sound out the survey. </li></ul></ul><ul><ul><li>For on-line, do technical runs to make sure it works on low res, works on slow bandwidth etc. </li></ul></ul>
  76. 81. <ul><li>Prepare a detailed outline of how you want the report to look, this is like a great insurance policy against the main mistakes that you can make in questionnaire design and scripting. It will ensure that you ask all of the relevant questions that are required to deliver the results and insights that you need. By and large, the challenges that you then face in constructing individual questions will be smaller as you will have a crystal clear idea of what you are trying to measure. </li></ul>Mark Solonsch Customer Insights and CTran Leader at GE Money “
  77. 82. Keep learning!
  78. 83. Expose yourself to feedback. <ul><li>For a listening profession, we don’t always listen. </li></ul><ul><li>At the end of questionnaires invite feedback – either a rating scale or open ended, or both. </li></ul><ul><li>This provides really good guidance, as well as ammunition to throw at cynics who say that nobody enjoys doing surveys. </li></ul>Actual scale I used on a long survey for feedback. Get a grip Duncan – this was not a good experience. This was a very engaging, interesting experience. 1 2 3 4 5 6 7 8 9 10
  79. 84. Expose yourself to feedback. <ul><li>“ Great survey, it is not until you do one of these you realise how much money you spend on things, made me feel good, with a tad of guilt, but you only live once i say, live it like its your last. Cheers!” </li></ul><ul><li> ” I love this style that you did, kept me going, wasn’t all formal and stiff (i.e. boring) GOOD JOB keep up the good work and I look forward to participating in another of your quiz things that aren’t quizzes soon :) </li></ul><ul><li>Thanks for giving me a taste of luxury on a day when I worked too hard to go out tonight! it was a little frisson for my soul. </li></ul><ul><li>Too long! </li></ul><ul><li>Interestingly presented survey. Didn’t go on too long with bulk &quot;matrix&quot; type questions e.g. what wine where and why. Images work well </li></ul><ul><li>A great simplified survey, keep up the good work. </li></ul>This is feedback after an on-line wine survey – average completion time 25-30 minutes.
  80. 85. Keep up your personal research. <ul><li>There are plenty of books on the subject. </li></ul><ul><li>There is limitless material on the web –including useful academic studies on questionnaire best practice. </li></ul><ul><li>There are professional chat groups where you can talk shop with colleagues. </li></ul><ul><li>Don’t stop. The mark of a professional researcher – of any professional for that matter – is the continuous challenge of the status quo, and the continuous belief that “better” is possible. </li></ul>
  81. 86. YOU THOUGHT WE’D NEVER GET HERE! Summary!
  82. 87. Summary <ul><li>Questionnaires are the single most difficult and riskiest task of the researcher. </li></ul><ul><li>Questionnaires must need the needs to several competing stakeholders - and that’s what makes writing them so tricky. </li></ul><ul><li>For the respondent we need to deliver an experience that respects where they’re at – they’re neither stupid or tolerant of being bored. </li></ul><ul><li>For the field team (phone or online or face to face) we need to deliver a document that acts as a well designed roadmap that doesn’t lead anyone astray or into grief. </li></ul><ul><li>For the analyst we need to design a document that generates data of the highest quality through questions that are relevant meaningful and answer-options that are crunch-able. </li></ul><ul><li>And the for the client we are designing a process that delivers measurement, guidance or inspiration – sometimes all three. </li></ul><ul><li>There are well established principles of questionnaire design that most of us regularly embrace – but there are also some well proven principles that may be new to us. </li></ul><ul><li>We owe it to ourselves as professionals to regularly listen to all our stakeholders, and to continue on our own quest to lift our game. By way of example, there are many, many researchers using “CATI Questionnaire” approaches to the online environment – and this is not necessarily desirable. </li></ul><ul><li>Finally – there is no one right way. A good questionnaire also contains your own DNA – your personal sense of style and character. Be yourself and keep working on it. </li></ul>
  83. 88. <ul><li>Construct questionnaire from general to specific - e.g. overall brand/concept ratings following by specific ratings of certain features, attributes. </li></ul><ul><li>Use simple language, no industry jargon. </li></ul><ul><li>Review each question to see if it assumes something on part of respondent - i.e. have done or seen something before, think a certain way... </li></ul><ul><li>Response options must be exhaustive and mutually exclusive </li></ul><ul><li>Response options must be balanced - i.e. can't have three positive response options and only two negative ones </li></ul><ul><li>Add &quot;or not&quot; to end of questions where it makes sense - Do you think this brand is hip or not? Does not dramatically alter the response distribution to not have &quot;or not&quot; but still worth adding. </li></ul><ul><li>Open debate on whether should have neutral response options (e.g. neither agree nor disagree) and it really does depend on circumstance </li></ul><ul><li>Also some open debate on whether should have &quot;don't know&quot; as explicit response option. For knowledge questions you certainly should, for opinion/attitude probably not but again, circumstance based. </li></ul><ul><li>No double barrel questions meaning asking two questions in one. Having &quot;and&quot; in a question is usually a dead give away but this can be very subtle. </li></ul>Kevin Schulman Senior Vice President at TRG iSky “
  84. 89. Duncan Stuart FMRSNZ <ul><li>2004 - elected as a Fellow of the MRSNZ. </li></ul><ul><li>Twice judge, NZ marketing Awards. </li></ul><ul><li>Five times best paper or people’s choice winner at MRSNZ Conferences. 1995, 97, 99, 01 and 03. </li></ul><ul><li>Extensive background in advanced analytics, text analysis and mixture of qual and quant approaches. </li></ul><ul><li>16 years in research. </li></ul><ul><li>2009 – elected as a Life Member of the MRSNZ </li></ul><ul><li>Telephone 366 0620 </li></ul><ul><li>[email_address] </li></ul><ul><li>www.kudos-dynamics.com </li></ul>
  85. 90. Thank you! Duncan Stuart FMRSNZ [email_address] Telephone 64 9 366 0620 www.kudos-dynamics.com If you find this presentation useful please support the language school we built and support in Siem Reap Cambodia . www.savong.com The website has PayPal.

×