Questionnaire design & admin


Published on

Published in: Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Questionnaire design & admin

  1. 1. Questionnaire Design & Administration Lecture 6
  2. 2. Chapter 9Questionnaire design 2
  3. 3. Learning Objectives• Discuss the process of questionnaire design• Draft, pre-test and revise questionnaires• Identify the potential sources of error that can occur when conducting a survey• Explain the issues involved in the administration of questionnaires using different survey methods• Discuss the ethical issues involved in collecting data from respondents
  4. 4. Questionnaire Design & Administration
  5. 5. A survey is only as good as the questions it asks• Questionnaire design is one of the most critical stages in the survey research process. – Importance of question wording is easily overlooked.• Relevance and accuracy are the two basic criteria for questionnaire design. 5
  6. 6. Questionnaires• Standardise data collection for survey• Ensure comparability of the data, increase speed and accuracy of recording and facilitate data processing.Definition• Formalised set of questions for obtaining information from respondents
  7. 7. Questionnaire design• Planning a questionnaire‟s design involves several decisions: – What should be asked? – How should questions be phrased? – In what sequence should the questions be arranged? – What questionnaire layout will best serve the research objectives? – How should the questionnaire be pre– tested? Does the questionnaire need to be revised? 7
  8. 8. What should be asked?• A questionnaire is relevant if no unnecessary information is collected and only the information needed to solve the marketing problem is obtained.• A questionnaire is accurate if the information obtained is reliable and valid. 8
  9. 9. How should questions be phrased?• Open–ended response questions pose some problem and ask the respondent to answer in his or her own words. – Most beneficial when conducting exploratory research.• Fixed–alternative questions give the respondent limited–alternative responses and ask him or her to choose one closest to his or her viewpoint. – Takes less time and easier to answer. 9
  10. 10. Objectives of a Questionnaire• Translate the information needed into a set of specific questions that the respondents can and will answer• To uplift, motivate, and encourage the respondent to become involved in the interview, to cooperate, and to complete the interview• Questionnaires should minimise response error
  11. 11. Questionnaire Administration• Personal interviews – Lengthy, complex and varied• Telephone interviews – Short and simple, conversational style• Mail surveys – Simple questions, detailed instructions
  12. 12. Questionnaire Design• Formulate questions to obtain the required information• Question order and wording• Sequence and layout• Pre-testing – correct and pre-test again Suggest questions/statement you could ask to measure: •Satisfaction with event/ establishment •Motivation to participate •Age •Determinant attributes
  13. 13. Formatting the Question• Open-response questions• Closed-response questions
  14. 14. Open-ended Response Questions• Open-response (unstructured) questions• Open-ended questions that respondents answer in their own words• Express general attitude• Interviewer bias is high• Coding is costly and time consumingWhat were you expecting by attending this event?
  15. 15. Open-ended Response Questions cont.• Used for: – introduction to a survey – when responses are unforeseen or there are too many to list – when verbatim responses are important – when behaviour being measured is sensitive
  16. 16. Closed Response Questions• Specify the set of response alternative and response format• Two alternative formats: – choice from a list of responses – choice on a rating scale
  17. 17. Closed Response Questions cont.• Advantages – Can be administered quickly – Coding and processing is less costly and time consuming – Easy to answer and easier for interviewer – Tabulation and analysis easier – Less potential error – Responses are directly comparable
  18. 18. Closed Response Questions cont.• Limitations – Effort required in designing alternatives (decision on type of responses) – May require exploratory research for determining choices (difficult to develop good questions) – Bias response because people pick alternatives offered (less ability for self- expression) – Potential order bias
  19. 19. Closed Response Questions cont.• Multiple-choice questions – Respondents are asked to select one or more alternative Which of the following people had an influence on your choice of university? None  Parents  Friends  Ex-Uni student  Teachers at secondary school  Careers teachers at secondary school  Colleagues  Other please specify…………………. 
  20. 20. Closed Response Questions cont.• Scales – Comparative and non-comparative (covered last week)• Dichotomous questions – Only two response alternatives – Mainly used for demographic and behavioural data Have you holidayed in Asia in the last 3 years? Yes No 
  21. 21. Question Wording• Vocabulary should be simple clear and direct• Words should not be vague or ambiguous• No „double-barreled‟ questions• No leading or loaded questions• Instructions should be clear• Respondents should only answer applicable questions• Test several alternatives of question wording• Questions should be appropriate length• Word sensitive questions carefully
  22. 22. Question Wording cont.Is the question necessary?• Every question should contribute to the information needed or serve some specific purpose.• Question wording should address the research objectives in order to make better decisions
  23. 23. Question Wording cont.Is the respondent informed?• Filter questions are often used to ascertain familiarity, product use and past experience – Are you the main grocery buyer in the household? – Have you seen the Holden Cruze ad? – Have you purchased car tyres in the last 3 months?• If respondents answer “No”, they may not be suitable respondents (i.e., they have not met the criteria for inclusion in the study)
  24. 24. Question Wording cont.Can the respondent remember?• Inability to remember can lead to – Errors of omission – Telescoping – Creation• The ability to remember events are influenced by – the event itself – time elapsed since the event – presence/absence of other events that would help memory
  25. 25. Question Wording cont.Can the respondent articulate?• Respondents may be unable to describe the situation accurately• Inability to articulate may lead to – respondents ignoring questions – refusing to respond to the rest of the questionnaire
  26. 26. Question Wording cont.Legitimate Purpose• Unwilling to divulge information which they do not see as serving a legitimate purpose• Specify why the question is being asked
  27. 27. Sequence and Layout DecisionsOpening Questions• Important in gaining confidence and co-operation of respondents• Questions should be interesting and non- threatening• Could be qualifying question
  28. 28. Sequence and Layout Decisions cont.Type of information Basic Classification Identification
  29. 29. Sequence and Layout Decisions cont.Difficult questions• Sensitive, embarrassing, complex or dull questions should be placed late in the sequenceEffect on subsequent question• Questions asked early in a sequence can influence the responses to subsequent questions• Use funnel or flowerpot approach
  30. 30. Sequence and Layout Decisions cont.Logical order• All questions that deal with a particular issue/topic should be asked before proceeding to a new topic• Use branching questions to direct respondents to other questions
  31. 31. Form and Layout• Format, spacing and position are important• A questionnaire can be divided into several parts• All questions should be numbered• Questionnaires are sometimes pre-coded where each possible response to a question is associated with a unique number (or letter)• Questionnaire should be serially numbered
  32. 32. What sort of strategies do you use to ensure high response rates in surveys? Michael Sparks, Director Intuitive Solutions
  33. 33. Reproduction of the Questionnaire• Use good quality paper and have a professional appearance• Consider using booklet if the questionnaire runs to several pages• Avoid splitting a question and response categories over two pages• Use single columns for response categories• Avoid overcrowding questions• Directions or instructions for individual questions should be placed close to the questions as possible e.g., [PROBING]
  34. 34. Pre-testing and Correcting Problems• Testing the questionnaire on a small sample of respondents to identify and eliminate potential problems• Use protocol analysis or debriefing• After each significant revision of the questionnaire, another pre-test should be performed with a different sample
  35. 35. Pre-testing• Debriefing approach – respondent completes questionnaire with interviewer observing – interviewer then debriefs respondent• Protocol approach – respondent asked to think aloud as they complete questionnaire – interviewer takes notes and asks for any further clarification at the end
  36. 36. Pre-testing cont.•Specific •TheQuestions Questionnaire –Variation –Sequence –Meaning –Form and layout –Task difficulty –Skip patterns –Respondent –Length interest and –Instructions attention
  37. 37. Cover Letter and Openers• Main role : to win over respondent‟s co-operation and willingness to participate• Respondents must be aware that a bonafide survey is taking place• Identify the sponsor• Purpose of survey should be described clearly and simply
  38. 38. Cover Letter and Openers cont.• Prospective respondents must be made aware of how and why they were selected• Assure respondent confidentiality• Specify the completion date• Cover letter should solicit the prospect respondent‟s help in the survey• Thank you statement for willingness to participate
  39. 39. Observational Forms• Specify the who, what when, where, why and way of behaviour to be observed• Offers consistency, structure, completeness
  40. 40. Administering the Questionnaire: Personal Interviews• Identification of location/venue• Recruitment of field staff – Develop job specifications for the project – Decide on the characteristics the fieldworkers should possess – Recruit appropriate individuals• Training of field staff – Conducted at central location or by mail – To ensure questionnaire is administered in the same manner
  41. 41. Training of Field StaffMaking the initial contact• To encourage respondents to participate• Convince potential respondents that their participation is important
  42. 42. Training of Field Staff cont.Asking the question• Be thoroughly familiar with the questionnaire• Ask the question in the order in which they appear• Use exact wording given in the questionnaire• Read questions slowly• Repeat questions that are not understood• Ask every applicable question• Follow instructions
  43. 43. Training of Field Staff cont.Probing• Motivate respondents to enlarge on, clarify, or explain their answers• Should not introduce any bias• How to probe?? – Repeat the question – Repeating the respondent‟s reply – Using a pause or silent probe – Boost or reassure respondents – Eliciting clarifications – Using objective / neutral questions or comments
  44. 44. Training of Field Staff cont.Recording the answers• Same format should be used• Structured Tick box that reflects the respondent‟s answer• Unstructured Record the responses verbatim – Record responses during the interview – Use the respondent‟s own words – Do not summarise or paraphrase the respondent‟s answers – Include everything that pertains to the question objectives – Include all probes and comments – Repeat the response as it is written down
  45. 45. Training of Field Workers cont.Terminating the Interview• Terminate after all information is obtained• Record spontaneous comments• Interviewer should answer the respondent‟s questions about the project• Thank respondent and express appreciation
  46. 46. Field Editing and CheckingQuality control and editing• Check that questions have been completed, do not accept incomplete answers, writing is legible• Keep record of hours worked and expenses
  47. 47. Field Editing and Checking cont.Sampling Control• Ensure interviews are strictly following sampling plan.• Keep record of: – number of calls made, – number of not-at-homes, – number of refusals, – number of completed interviews per interviewer.Control of cheating• Falsifying part of a question or entire questionnaire.• Cheating can be minimised through proper training, supervision and validation of field work.
  48. 48. Field Editing and Checking cont.Validation• Check for authenticity.• 10 – 25% of respondents are contacted.• Ask about length and quality of the interview, reaction to the interviewer, and basic demographic data.
  49. 49. Field Editing and Checking cont.Evaluation• Provide feedback to interviewers on: – Cost and time – Response rates – Quality of interviewing – Quality of data
  50. 50. Telephone Interviews• Similar to personal surveys plus: – selecting the telephone numbers – call outcomes – the introduction – when to call – [6-8pm week days, 10-8 weekends] – Call report
  51. 51. Mail Surveys• Identification of respondents• Mail interview package – Outgoing envelope – Reply paid envelope – Postage – Method of addressing – Cover letter – Questionnaire length, content, layout etc – Method of notification – Incentives – Follow up
  52. 52. Online Surveys– Similar to mail surveys– Design the cover letter/email– Technology issues should be addressed
  53. 53. Salaries and WagesClassification Per hour rate*Market Research Interviewer $21.72Door-to Door $23.72Editor/Coder/Key Punch Operator $22.12Field Supervisor $26.32Research Assistant $26.32Research Officer $29.58Research Manager $42.20 * Casual rates (Australian dollars) Source: NUW and AMRO (2005) Report on the Market Research Industry Agreement 2005 - 2008
  54. 54. Non-response Issues• Non-response error arises when potential respondents included in the sample do not respond.• Major problem is whether non-respondents differ from respondents in terms of demographic, psychographic, personality, attitudinal, motivational and behavioural variables.• To reduce non-response bias: improve the response rate and/or adjust for non- response.
  55. 55. Improving Response Rates
  56. 56. Sources of Error• Inaccuracy in response – inability to respond – unwillingness to respond• Interviewer error – lack of interviewing skills – fraud and deceit
  57. 57. Unethical Practices in Data Collection• Using of surveys as a disguise for selling purposes• Using details in survey to develop a prospect list• Disguising the purpose of the measurement• Deceiving the respondent as to the true duration of the interview• Misrepresenting the compensation in order to obtain co-operation• Not informing respondents of follow-up interviews• Using hidden taping without respondent‟s consent• Conducting simulated product test in which identical products are given• Not debriefing the respondent
  58. 58. Types of fixed–alternative questions• Simple–dichotomy: choose one of two alternatives.• Determinant–choice: choose one and only one from among several alternatives.• Frequency–determination: asks for general frequency of occurrence.• Checklist: choose multiple answers to a single question. 59
  59. 59. Phrasing questions for surveys• Means of data collection will influence the question format and phrasing. – Questions for mail, Internet, and telephone surveys are less complex than those used in personal interviews. – Questionnaires for telephone and personal interviews should be written in a conversational style. 60
  60. 60. The art of asking questions• Some guidelines to help prevent the most common mistakes: – Avoid complexity: use simple, conversational language. – Avoid leading and loaded questions. – Avoid ambiguity: be as specific as possible. – Avoid double–barrelled items. – Avoid making assumptions. – Avoid burdensome questions that may tax the respondent‟s memory. 61
  61. 61. What is the best question sequence?• Order of questions serve several functions. – Opening questions influence respondents‟ cooperation and involvement. – Demographic questions that may embarrass or threaten respondents should be asked at the end of the questionnaire.• Order bias may distort survey results. – Funnel technique: asking general questions before specific questions. 62
  62. 62. What is the best question sequence?• A filter question screens out respondents who are not qualified to answer a second question. – A pivot question is used to determine which version of a second question will be asked. 63
  63. 63. What is the best layout?• Layout should be neat, attractive and interviewer instructions easy to follow. – Decent margins, white space to separate blocks of print, etc.,• Should appear as short as possible. – Multiple–grid question• Layout is also important for Internet questionnaires. – Discrepancies in screen configuration, push button, status bar, radio button, etc., 64
  64. 64. What is the best layout? 65
  65. 65. What is the best layout? 66
  66. 66. How much pre–testing and revising are necessary?• Questionnaire is usually tried out on a group that is similar in make–up to the ultimate sample.• A preliminary tabulation of pre–test results helps determine whether the questionnaire will meet the objectives of the research.• Pre–tests provide means for the testing sampling procedure. 67
  67. 67. How much pre–testing and revising are necessary?• Pre–tests typically are conducted to answer the following questions about the questionnaire: – Can the format be followed by interviewer? – Does it flow naturally and conversationally? – Are the questions clear and easy to understand? – Can respondents answer the questions clearly? – Which alternative forms of questions work best? 68
  68. 68. Designing questionnaires for global markets• Researchers must take cultural factors into account when designing questionnaires.• The most common problem involves translating a questionnaire into other languages. – International marketing researchers often have questionnaires back translated. 69