• Save
How to design questionnaire
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

How to design questionnaire

  • 9,566 views
Uploaded on

How to design questionnaire, B R M

How to design questionnaire, B R M

More in: Education , Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • any one have questions for this passage, could you send they for me please.

    yabandi@hotmail.com
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
9,566
On Slideshare
9,566
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
1
Likes
18

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. HOW TO DESIGN QUESTIONNAIRE – BRMREPORTQuestionnaire DesignQuestionnaires are an inexpensive way to gather data from a potentially large number ofrespondents. Often they are the only feasible way to reach a number of reviewers large enoughto allow statistically analysis of the results. A well-designed questionnaire that is used effectivelycan gather information on both the overall performance of the test system as well as informationon specific components of the system. If the questionnaire includes demographic questions onthe participants, they can be used to correlate performance and satisfaction with the test systemamong different groups of users.It is important to remember that a questionnaire should be viewed as a multi-stage processbeginning with definition of the aspects to be examined and ending with interpretation of theresults. Every step needs to be designed carefully because the final results are only as good asthe weakest link in the questionnaire process. Although questionnaires may be cheap toadminister compared to other data collection methods, they are every bit as expensive in termsof design time and interpretation.The steps required to design and administer a questionnaire include: 1. Defining the Objectives of the survey 2. Determining the Sampling Group 3. Writing the Questionnaire 4. Administering the Questionnaire 5. Interpretation of the ResultsThis document will concentrate on how to formulate objectives and write the questionnaire.Before these steps are examined in detail, it is good to consider what questionnaires are good atmeasuring and when it is appropriate to use questionnaires.What can questionnaires measure?Questionnaires are quite flexible in what they can measure, however they are not equally suitedto measuring all types of data. We can classify data in two ways,Subjective vs.Objective and Quantitative vs. Qualitative.When a questionnaire is administered, the researchers control over the environment will besomewhat limited. This is why questionnaires are inexpensive to administer. This loss of controlmeans the validity of the results are more reliant on the honesty of the respondent.
  • 2. Consequently, it is more difficult to claim complete objectivity with questionnaire data then withresults of a tightly controlled lab test. For example, if a group of participants are asked on aquestionnaire how long it took them to learn a particular function on a piece of software, it islikely that they will be biased towards themselves and answer, on average, with a lower thanactual time. A more objective usability test of the same function with a similar group ofparticipants may return a significantly higher learning time. More elaborate questionnairedesign or administration may provide slightly better objective data, but the cost of such aquestionnaire can be much higher and offset their economic advantage. In general,questionnaires are better suited to gathering reliable subjective measures, such as usersatisfaction, of the system or interface in question.Questions may be designed to gather either qualitative or quantitative data. By their very nature,quantitative questions are more exact then qualitative. For example, the word "easy" and"difficult" can mean radically different things to different people. Any question must be carefullycrafted, but in particular questions that assess a qualitative measure must be phrased to avoidambiguity. Qualitative questions may also require more thought on the part of the participantand may cause them to become bored with the questionnaire sooner. In general, we can say thatquestionnaires can measure both qualitative and quantitative data well, but that qualitativequestions require more care in design, administration, and interpretation.When to use a questionnaire?There is no all encompassing rule for when to use a questionnaire. The choice will be madebased on a variety of factors including the type of information to be gathered and the availableresources for the experiment. A questionnaire should be considered in the followingcircumstances.a. When resources and money are limited. A Questionnaire can be quite inexpensive toadminister. Although preparation may be costly, any data collection scheme will have similarpreparation expenses. The administration cost per person of a questionnaire can be as low aspostage and a few photocopies. Time is also an important resource that questionnaires canmaximize. If a questionnaire is self-administering, such as a e-mail questionnaire, potentiallyseveral thousand people could respond in a few days. It would be impossible to get a similarnumber of usability tests completed in the same short time.b. When it is necessary to protect the privacy of the participants.Questionnaires areeasy to administer confidentially. Often confidentiality is the necessary to ensure participantswill respond honestly if at all. Examples of such cases would include studies that need to askembarrassing questions about private or personal behavior.c. When corroborating other findings. In studies that have resources to pursue other data
  • 3. collection strategies, questionnaires can be a useful confirmation tools. More costly schemesmay turn up interesting trends, but occasionally there will not be resources to run these othertests on large enough participant groups to make the results statistically significant. A follow-uplarge scale questionnaire may be necessary to corroborate these earlier results.I. Defining the Objectives of the SurveyThe importance of well-defined objectives can not be over emphasized. A questionnaire that iswritten without a clear goal and purpose is inevitably going to overlook important issues andwaste participants time by asking useless questions. The questionnaire may lack a logical flowand thereby cause the participant to lose interest. Consequential, what useful data you may havecollected could be further compromised. The problems of a poorly defined questionnaire do notend here, but continue on to the analysis stage. It is difficult to imagine identifying a problemand its cause, let alone its solution, from responses to broad and generalizing questions. In otherwords, how would it be possible to reach insightful conclusions if one didnt actually know whatthey had been looking for or planning to observe.A objective such as "to identify points of user dissatisfaction with the interface and how thesenegatively affect the softwares performance" may sound clear and to the point, but it is not. Thequestionnaire designer must clarify what is meant by user dissatisfaction. Is this dissatisfactionwith the learning of the software, the power of the software, of the ease of learning the software?Is it important for the users to learn the software quickly if they learn it well? What is meant bythe softwares performance? How accurate must the measurements be? All of these issues mustbe narrowed and focused before a single question is formulated. A good rule of thumb is that ifyou are finding it difficult to write the questions, then you havent spent enough time definingthe objectives of the questionnaire. Go back and do this step again. The questions should followquite naturally from the objectives.II. Writing the QuestionnaireAt this point, we assume that we have already decided what kind of data we are to measure,formulated the objectives of the investigation, and decided on a participant group. Now we mustcompose our questions.If the preceding steps have been faithfully executed, most of the questions will be on obvioustopics. Most questionnaires, however, also gather demographic data on the participants. This is
  • 4. used to correlate response sets between different groups of people. It is important to seewhether responses are consistent across groups. For example, if one group of participants isnoticeably less satisfied with the test interface, it is likely that the interface was designed withoutfair consideration of this groups specific needs. This may signify the need for fundamentalredesign of the interface. In addition, certain questions simply may only be applicable to certainkinds of users. For example, if one is asking the participants whether they find the new tutorialhelpful, we do not want to include in our final tally the responses of experienced users wholearned the system with an older tutorial. There is no accurate way to filter out these responseswithout simply asking the users when they learned the interface.Typically, demographic data is collected at the beginning of the questionnaire, but suchquestions could be located anywhere or even scattered throughout the questionnaire. Oneobvious argument in favor of the beginning of the questionnaire is that normally backgroundquestions are easier to answer and can ease the respondent into the questionnaire. One does notwant to put off the participant by jumping in to the most difficult questions. We are all familiarwith such kinds of questions.It is important to ask only those background questions that are necessary. Do not ask income ofthe respondent unless there is at least some rational for suspecting a variance across incomelevels. There is often only a fine line between background and personal information. You do notwant to cross over in to the personal realm unless absolutely necessary. If you need to solicitpersonal information, phrase your questions as unobtrusively as possible to avoid ruffling yourparticipants and causing them to answer less than truthfully.What kind of questions do we ask?In general, there are two types of questions one will ask, open format or closed format.Open format questions are those that ask for unprompted opinions. In other words, there are nopredetermined set of responses, and the participant is free to answer however he chooses. Openformat questions are good for soliciting subjective data or when the range of responses is nottightly defined. An obvious advantage is that the variety of responses should be wider and moretruly reflect the opinions of the respondents. This increases the likelihood of you receivingunexpected and insightful suggestions, for it is impossible to predict the full range of opinion. Itis common for a questionnaire to end with and open format question asking the respondent forher unabashed ideas for changes or improvements.Open format questions have several disadvantages. First, their very nature requires them to beread individually. There is no way to automatically tabulate or perform statistical analysis onthem. This is obviously more costly in both time and money, and may not be practical for lowerbudget or time sensitive evaluations. They are also open to the influence of the reader, for no
  • 5. two people will interpret an answer in precisely the same way. This conflict can be eliminated byusing a single reader, but a large number of responses can make this impossible. Finally, openformat questions require more thought and time on the part of the respondent. Whenever moreis asked of the respondent, the chance of tiring or boring the respondent increases.Closed format questions usually take the form of a multiple-choice question. They are easy forthe respondent, giveThere is no clear consensus on the number of options that should be given in an closed formatquestion. Obviously, there needs to be sufficient choices to fully cover the range of answers butnot so many that the distinction between them becomes blurred. Usually this translates into fiveto ten possible answers per questions. For questions that measure a single variable or opinion,such as ease of use or liability, over a complete range (easy to difficult, like to dislike),conventional wisdom says that there should be an odd number of alternatives. This allows aneutral or no opinion response. Other schools of thought contend that an even number ofchoices is best because it forces the respondent to get off the fence. This may induce the someinaccuracies for often the respondent may actually have no opinion. However, it is equallyarguable that the neutral answer is over utilized, especially by bored questionnaire takers. Forlarger questionnaires that test opinions on a very large number of items, such as a music test, itmay be best to use an even number of choices to prevent large numbers of no-thought neutralanswers.Closed format questions offer many advantages in time and money. By restricting the answerset, it is easy to calculate percentages and other hard statistical data over the whole group orover any subgroup of participants. Modern scanners and computers make it possible toadminister, tabulate, and perform preliminary analysis in a matter of days. Closed formatquestions also make it easier to track opinion over time by administering the samequestionnaire to different but similar participant groups at regular intervals. Finally closedformat questions allow the researcher to filter out useless or extreme answers that might occurin an open format question.Whether your questions are open or closed format, there are several points that must byconsidered when writing and interpreting questionnaires: 1. Clarity: This is probably the area that causes the greatest source of mistakes in questionnaires. Questions must be clear, succinct, and unambiguous. The goal is to eliminate the chance that the question will mean different things to different people. If the designers fails to do this, then essentially participants will be answering different questions.
  • 6. To this end, it is best to phrase your questions empirically if possible and to avoid the use ofnecessary adjectives. For example, it asking a question about frequency, rather than supplyingchoices that are open to interpretation such as: 1. Very Often 2. Often 3. Sometimes 4. Rarely 5. NeverIt is better to quantify the choices, such as: 1. Every Day or More 2. 2-6 Times a Week 3. About Once a Week 4. About Once a Month 5. NeverThere are other more subtle aspects to consider such as language and culture. Avoid the use ofcolloquial or ethnic expressions that might not be equally used by all participants. Technicalterms that assume a certain background should also be avoided. 1. Leading Questions: A leading question is one that forces or implies a certain type of answer. It is easy to make this mistake not in the question, but in the choice of answers. A closed format question must supply answers that not only cover the whole range of responses, but that are also equally distributed throughout the range. All answers should be equally likely. An obvious, nearly comical, example would be a question that supplied these answer choices: 1. Superb 2. Excellent 3. Great 4. Good 5. Fair 6. Not so GreatA less blatant example would be a Yes/No question that asked:
  • 7. 1. Is this the best CAD interface you have every used?In this case, even if the participant loved the interface, but had an favorite that was preferred,she would be forced to answer No. Clearly, the negative response covers too wide a range ofopinions. A better way would be to ask the same question but supply the following choices: 1. Totally Agree 2. Partially Agree 3. Neither Agree or Disagree 4. Partially Disagree 5. Totally AgreeThis example is also poor in the way it asks the question. Its choice of words makes it a leadingquestion and a good example for the next section on phrasing. 1. Phrasing: Most adjectives, verbs, and nouns in English have either a positive or negative connotation. Two words may have equivalent meaning, yet one may be a compliment and the other an insult. Consider the two words "child-like" and "childish", which have virtually identical meaning. Child-like is an affectionate term that can be applied to both men and women, and young and old, yet no one wishes to be thought of as childish.In the above example of "Is this the best CAD interface you have every used?" clearly "best" hasstrong overtones that deny the participant an objective environment to consider the interface.The signal sent the reader is that the designers surely think it is the best interface, and so shouldeveryone else. Though this may seem like an extreme example, this kind of superlative questionis common practice.A more subtle, but no less troublesome, example can be made with verbs that have neitherstrong negative or positive overtones. Consider the following two questions: 1. Do you agree with the Governors plan to oppose increased development of wetlands? 2. Do you agree with the Governors plan to support curtailed development of wetlands?They both ask the same thing, but will likely produce different data. One asks in a positive way,and the other in a negative. It is impossible to predict how the outcomes will vary, so one
  • 8. method to counter this is to be aware of different ways to word questions and provide a mix inyour questionnaire. If the participant pool is very large, several versions may be prepared anddistributed to cancel out these effects.4. Embarrassing Questions: Embarrassing questions dealing with personal or privatematters should be avoided. Your data is only as good as the trust and care that your respondentsgive you. If you make them feel uncomfortable, you will lose their trust. Do not askembarrassing questions. 1. Hypothetical Questions Hypothetical are based, at best, on conjecture and, at worst, on fantasy. I simple question such as: 1. If you were governor, what would you do to stop crime?This forces the respondent to give thought to something he may have never considered. Thisdoes not produce clear and consistent data representing real opinion. Do not ask hypotheticalquestions. 1. Prestige Bias: Prestige bias is the tendency for respondents to answer in a way that make them feel better. People may not lie directly, but may try to put a better light on themselves. For example, it is not uncommon for people to respond to a political opinion poll by saying they support Samaritan social programs, such as food stamps, but then go on to vote for candidates who oppose those very programs. Data from other questions, such as those that ask how long it takes to learn an interface, must be viewed with a little skepticism. People tend to say they are faster learners than they are.There is little that can be done to prevent prestige bias. Sometimes there just is no way to phrasea question so that all the answers are noble. The best means to deal with prestige bias is to makethe questionnaire as private as possible. Telephone interviews are better than person-to-personinterviews, and written questionnaires mailed to participants are even better still. The fartheraway the critical eye of the researcher is, the more honest the answers.Now What?Now that youve completed you questionnaire, you are still not ready to send it out. Just like anymanufactured product, your questionnaire needs to go through quality testing. The majorhurdle in questionnaire design is making it clear and understandable to all. Though you havetaken great care to be clear and concise, it is still unreasonable to think that any one person can
  • 9. anticipate all the potential problems. Just as a usability test observes a test user with the actualinterface, you must observe a few test questionnaire takers. You will then review thequestionnaire with the test takers and discuss all points that were in any way confusing andwork together to solve the problems. You will then produce a new questionnaire. It is possiblethat this step may need to be repeated more than once depending on resources and the need foraccuracy.QUESTIONNAIRE DESIGN, SURVEY METHODS, AND THE ART OFASKING QUESTIONSQuestionnaires are used in connection with many modes of observation in social research. Theyserve as an essential tool in public opinion survey research. The format of a questionnaire is asimportant as the wording of the questions asked. An improperly designed questionnaire canlead respondents to skip questions, confuse respondents as to the nature of the data desired, andin the extreme, lead people to refuse to respond.I. Standardized QuestionnairesA questionnaire is like a conversation with a purpose. It is important to ask initially, do I want touse a standardized questionnaire? A standardized questionnaire is one in which everyrespondent gets the same set of questions in the same order. Typically, if a large sample isinvolved, the services of many interviewers required, and the data are to be analyzedstatistically. A standardized questionnaire is the preferred methodology.II. Deciding on ContentConsiderations in constructing a questionnaire:1. Decide what information is required to produce independent, dependent and controlvariables.2. Draft some questions to gather that information.3. Put them into meaningful order and format (e.g. open ended, closed ended, Likert)4. Pretest the result on a sample of potential respondents.5. Make necessary changes to the final form of the questionnaire.III. Understanding the Dimensions of Opinion.There are a variety of dimensions of public opinion according to George Gallup they are:1. A persons knowledge or awareness of the issue is a major dimension. People can haveopinions that have no basis in facts.2. A persons interest or the saliency of the issue to the person is an important dimension. Somepeople are apathetic others highly concerned about a given issue; e.g. gun control.
  • 10. 3. A persons attitude or the direction of his opinion is another dimension. (Typically, for oragainst an issue).4. A persons strength of believe- from very strong to slightly is another dimension.5. Why a person believes as he or she does, this is the most difficult question to answer.III. Mode of AdministrationThere are three major ways to administer questionnaire. Mail. Face to Face, Telephone.A) Mail Respondents receive questionnaire, read it and administer it to them elves (fill it out).Good for collecting data from specialized and highly motivated groups for example, planners.Very cost effective since one of the main costs in survey research is locating the respondent andinterviewer costs. You can get both wide geographical spread and cost savings from mailquestionnaires, it costs no more to do a national than a statewide survey of the same size...a 29cent stamp costs the same.Disadvantages1. You dont have rapport with respondent.2. No probing for more specific responses from respondents is possible.3. The timeliness of mail questions does not make them appropriate for dynamic issues, that isthings subject to rapid change.4. Question order bias may emerge because respondent can see the entire questionnaire beforeanswering the first question. This is a real problem if the order in which the questions are to beasked is important to the research design.5. You are never sure if the respondent sampled is the one who filled out the questionnaire andless educated people may have trouble following instructions and there is no one to explainthem.6. If the question asks information questions and you want to measure knowledge therespondent has the opportunity to look up the answers. It is difficult to measure knowledge.7. There are no enumerated lists from which to sample of the general public.8. You can get low response rates without inducements.9. You cant display or present certain alternatives that you can in a face to face setting; e.g. thisis our product which of these three package designs do you prefer?10. The size of the questionnaire is more of a limitation and you compete with junk mail or if youare doing the survey online with Spam.Personal Interviews Personal interviews are conducted in one of two ways face to face or byphone. They are extremely expensive and require skilled interviewers who must be trained andsupervised. There are certain types of questions that can only be dealt with using this technique,for example, teenage sexuality, which is a very controversial issue. This involves the interviewer
  • 11. asking questions in the physical presence of the respondent. It is an effective tool when thelength of the interview exceeds 20 minutes. It is extremely effective when you want to display aproduct or device; e.g. photographs of potential candidates or products. Some elements of thepopulation can be reached only by going to the door or in a given location; e.g. delegates to theRepublican convention. You have the rapport factor not possible by mail or phone. Increasingthe computer is used to administer and collect information about sensitive issues. Typically, amultimedia approach is employed in which a questionnaire is programmed to ask and capturethe answers of thew respondent without a human being present. RTI has pioneered some ofthese applications.Disadvantages1. Face to face interviewing allows for greater control over the respondents behavior, but canresult in interviewer respondent interactions like white interviewer asking black respondentabout L.A. riots get different responses than black interviewers.2. Very costly you need to locate and interview respondents.3. You cant control interviewer bias (what they say how they follow instructions) because theyare out of your span of control.4. Many respondents are in hard to reach places, behind closed doors, or, in undesirable parts ofthe city.C) TelephoneTelephone interviewing is an excellent method for timely generalizations. With a CATI system,your results are right there after the interview. Telephone interviews are excellent for generalaudiences and have a large geographical reach. They are cheaper than a personal interviewconducted on a face to face basis and have completion rates that are similar. Studies show thatthe quality of data collected by telephone surveys is comparable to other methods. Finally, interms of interviewer effects they are free of many of the biases that are possible in face to faceinterviewing. Also, with random digit dialing and computer sampling of phone lists, you cantake random samples of households even those phone is unlisted.Disadvantages1. They are disruptive of home life and are easier to reject.2. They compete with gimmicks that pretend to be surveys when they are really phonesolicitation. Many legitimate polls now start with the statement that we will not attempt to sellyou anything....3. Like mail questionnaires certain forms of presentation are more difficult or impossible.4. It under represents those without phones. About 95% of all households have them.
  • 12. 5. Answering machines can act as a screen to prevent the interviewer from direct contact withthe respondent.IV. Survey ProcessNote: you need to have a good understanding of the following:1. What are your independent variables? What do you believe is the variable that is influencingother variables? If you believe college educated people are more likely than non college educatedpeople to buy your product or vote for your candidate, then education is the independentvariable.2. What are your dependent variable(s)? This is the variable being effected, In the aboveexample it is the product or the candidate.3. What are your control variables? These are variables that need to be held constant in order tomeasure the effect of the IV on the DV. For example, a control, variable might be race, in whichcase, you would want to look at the relationship between education and candidate choice (vote)for both Afro-Americans and Whites separately so that you can control (hold constant) the effectof race.4. At what level of measurement is very variable in your study measured, nominal, and ordinal.Interval, ratio. Why? All things being equal measure at the highest level of measurementpossible?5. What are the hypotheses you are testing? Remember that a hypothesis is a relationshipbetween two or more variables. For example; mothers with children in this study are more liketo support anti pornography legislation than other women or the men in the sample.6. Do you have single or multiple measurements (questions) of your concepts e.g. conservatismversus liberalism, subjective or objective measures, one or multiple questions.V. Questionnaire FormatA. Question Sequence.IntroductionMain BodyDemographicsIntroduction. It should be short, realistically worded, non-threatening, serious, neutral andestablish legitimacy. It does not necessarily have to ask permission. It helps today to add thatyou will attempt to sell them nothing. Also, this is the point at which you will do any screening,e.g. ASK REGISTERED VOTERS ONLY.Main Body. This is typically the main body of IVs DVs Control Variables. There is a majordivision of the types of question based on weather the question can be verified (fact orempirical) These are behavioral or empirical questions and are verifiable. Opinion questions,
  • 13. These ask the respondent to rank with some degree of intensity statements designed to placethem on a continuum with respect to a concept.Demographics. These should be placed at the back of the questionnaire. They typicallyconstitute the major independent variables of the study.B. Open and Closed Ended QuestionsThere are two options in the realm of asking questions: open-ended and closed-ended questions.Whereas in open-ended questions respondents are asked to provide their own answers to thequestion, in closed-ended questions they are asked to select their answer from among a listprovided to them. Closed-ended questions yield greater uniformity of responses and are moreeasily processed, thus they are preferred by most researchers. However, in terms of reliabilityand validity, closed-ended questions must provide response categories that are exhaustive andmutually exclusive. To minimize form differences, investigators are advised to pretest theirquestionnaire. They often begin with open questions on large samples of the target populationand use the responses to construct closed alternatives that reflect the substance and wording ofwhat people say spontaneously. Despite the obvious advantages of closed questions, openquestions cannot be eliminated from survey research.Open-ended questions may be needed:- to pretest closed categories- to document the absence of a type of response- to follow up on closed questions and determine why people answer the way they do- to gather data where rapidly shifting external events can affect answers- to avoid newly emerging categories (over a long time period)- if the set of meaningful alternatives is too large or complex to present to respondentsC. StatementsAlthough survey research is commonly viewed as the asking of questions, researchers are ofteninterested in the extent to which respondents hold a particular attitude or position. Such aposition is often summarized in a statement, a declarative sentence which respondents are askedto agree or disagree with. This procedure is formalized through the creation of a Likert scale, aformat in which respondents are asked to "strongly agree", "agree", "disagree", "stronglydisagree", or "strongly approve", "strongly disapprove", etc ...D. Semantic DifferentialThis is a device in which an person, object, office, etc. is evaluated by list a series of bipolaradjectives with points of gradations in between. The respondent can then rank the ideacongressman and then two real congressmen. The research can then compare differences.E. Skipping and Filter QuestionsOccasionally it is necessary to skip some questions and branch to other questions depending
  • 14. upon the respondents answer.For example, ASK ONLY IF RESPONDENT IS REGISTERED TO VOTE, may be a filter forquestions about voting preferences.VI. Importance of Response RateIn general, using inferential statistics in connection with survey analysis assumes that allmembers of the initial sample complete and return their questionnaire. Problems associatedwith low response rates arise because the researcher intends to generalize survey information toa group larger than that from which the data were obtained. It may be that nonrespondentsdiffer systematically from respondents and this difference is not reflected in the data obtained.The purpose for indicating survey response rates is to help the reader decide to what extent thesample data can be generalized to the whole population. Three main reasons for nonresponsesare:(a) selected sample members never receive the questionnaire;(b) they refuse to answer; and(c) they forget to complete or return the survey.The higher the response rate the better the more likely the survey is representative. As a rule ofthumb, a response rate of 50 percent is considered adequate, 60 percent good, and 70 percent ormore very good. There are techniques that can increase response rate.Empirical research has demonstrated evidence that the following techniques can help toimprove response rates to mail questionnaires:Most Effective:- use of follow-ups (especially telephone reminders and special delivery letters)- pre-contact with respondents- type of postage (special delivery and airmail are superior to ordinary mail)- cash rewards- appearance of the sponsoring organization and title of the person signing the accompanyingletter (the more prestigious the better)Somewhat Effective:- explaining the place and importance of the respondent for the survey- personalizing the questionnaire- Anonymity- Appeals to the respondent based on the social benefit of the studyVII. The Art of Asking QuestionsQuestionnaires are structured conservation with a purpose. The format of a questionnaire is as
  • 15. important as the nature and wording of the questions asked. An improperly designedquestionnaire can lead respondents to miss questions, confuse them as to the nature of the datadesired, and in the extreme, lead them to refuse to respond. The purpose of asking questions ina survey is to elicit information from the respondent. Some general rules should be followed inthe design and wording of questions and statements.1. Making Items clear: Questionnaire items must be clear and unambiguous. Theyshould be precise so that the respondent knows exactly what question he/she isexpected to answer.2. Avoiding Double-Barreled Questions: Never ask for a single answer to acombination of questions. In general, whenever the words and or appear in aquestion or questionnaire statement, check whether this is a double-barreledquestion.3. Asking Relevant Questions: Questions asked in a survey should be relevant tomost respondents. Disregard responses to fictitious issues.4. Using short Items: Assume that respondents read items quickly and providequick answers. Therefore, provide clear, short items that will not bemisinterpreted under such conditions.5. Avoiding negative Terms: Negations (e.g. the use of the word "not") will often isread over, leading to misinterpretations.6. Avoiding biased Items and Terms: The manner in which data are soughtdetermines the nature of the data received. The identification of an attitude orposition with a prestigious or popular person, for example, can bias responses(e.g. President Clinton vs. Adolf Hitler).7. Avoid the Ambiguity of Language When we say that there is an art to askingquestions that give unbiased answers we mean exactly that. There are rules thatyou can follow, but there is an art that comes with practice and trial and error toasking questions fairly. Here are some things to avoid.VIII. Examples of Bias in Questions:Unfair AlternativesBad: Some people say that the city is spending too much on building new public schools. Do youagree or disagree?Improved: Some people say that the city is spending too much on building public schools ... andothers say the city is not spending enough. With which opinion do you agree?Maligning the Other SideBad: Do you think the government should spend more of our tax money on the slums?
  • 16. Improved: Do you think the government should spend more ... or less money on replacing theslum neighborhoods in the city with new housing projects?Damning with Faint PraiseBad: Some people say that the Mayors plan is a poor plan to solve garbage removal problems inthe city. Others say it will do for now until a better solution is found. Do you think it is a goodplan or a poor plan?Improved: Some people favor and some oppose the plan for combined garbage and trashremoval by the city.Do you think the plan is a good solution ... or a poor solution to the garbage removal problem?Deliberately Omitting NamesBad: Hello ... Im conducting a poll for Sam Snide, a candidate for mayor of the city. If theelection were held today, whom would you vote for ... Mr. Snide, or one of the other candidates?Improved: I am conducting a survey on the mayoral election...Inappropriate Use of TitlesBad: State Attorney General Allen P. Mutt is running for governor this year against Tom L. Jeff.Which man, Mutt or Jeff, is best qualified to be governor?Improved: Allen P Mutt and Tom L. Jeff. Are running for governor this year. Which man, Muttor Jeff, is best qualified to be governor?PersonalitiesBad: Would you say that governor Hunts energy program for promoting solar heating of privatehomes has been very effective, fairly effective, not too effective, or not effective at all?Improved: Would you say that the state energy program for promoting solar heating of privatehomes has been very effective, fairly effective, not too effective, or not effective at all?Emotionally Charged WordsBad: Congressman Pork barrel has been accused of defrauding the voters of this district. Do youagree or disagree with that charge?Improvement: One of the issues in this campaign is how well Congressman Pork barrel hascarried out his campaign promises. Do you think that Pork barrel has done an excellent, good,poor, or very poor job of doing what he said he would do?Conditioned by ContextQ1: Tom Fetzer is mayor of the city. In your opinion is he doing a good ... or poor job as mayor?Q2: As you understand it, what are the mayors principal duties in office?Note: the response to the second question is conditioned by your response to the first it wouldbe better to reverse the order or ask only one!Embarrassing QuestionsBad: How much time did you spend reading the newspaper yesterday?
  • 17. Improvement: Did you have a chance to read the newspaper yesterday? (IF YES: About howmuch time did you spend reading the newspaper yesterday?)Bad: What is your religion?Improvement Do you happen to have a religious preference? (IF YES: What is your religiouspreference?)Bad: Did you vote in the city election last month?Improvement Did you happen to vote in the city election last month, or didnt you have a chanceto vote?Bad: How old are you?Improvement: In what year were your born?Illogical Sentence ConstructionBad: Some people say that Senator Helms is doing an excellent job in office, and some peoplesay he is doing a very poor job. What kind of job do you think Senator Helms is doing ...excellent, good, poor, or very poor?Improvement: Would you say that Helms is doing an excellent, good, poor, or very poor job asUnited States Senator?Two-part QuestionsBad: Do you think that Mayor Booth should run for re-election this year, or could the Democratsfind a stronger candidate?Improvement: Q1: Do you think that Mayor Booth should or should not run for re-election thisyear?Q2: Do you think the Democrats could or could not find a stronger candidate than Mayor Booththis year?Ambiguous QuestionsBad: Did you vote in the last election?Improvement: Did you vote in the city election for Mayor last June?Bad: Are you in favor of a larger government role in housing and the environment?Improvement: Q1: Are you in favor of a larger role for the federal government in theenvironment? Q2: Are you in favor of a larger role for the federal government in housing?Note: If you have "and" in the question with two possible responses, and the respondent has adifferent view of the federal governments role in housing versus the environment he/she cantanswer the question.Indefinite Persons or PlacesBad: Are there many voters living around here?Improvement: Of the people you personally know living on Apple Street between 34th and 35thAvenues, about how many do you know to be registered to vote ... Would you say nearly all,
  • 18. maybe about three-quarters, about half ... or less than half?Indefinite ConceptsBad: Among you circle of friends, is there anyone whose opinions or advice you frequently askabout the public affairs issues of the day?Improvement: Among your circle of friends is there anyone whose opinions or advice youfrequently ask ... about such issues as the energy crisis?Misperceived Questions1. Outside Respondents ExperienceInterviewer asked: Generally, do you think that precinct captains get involved in politics foregocentric or altruistic reasons?Respondent said: "I dont care what their financial reasons are, policemen should keep out ofpolitics."2. Technical wordsInterviewer asked: Do you think we should change the present system of the Electoral College?Respondent said: "No, the universities are all right. Its just one bunch of leftwing kids makingall the trouble."Interviewer asked: Do you favor giving more power to election judges?Respondent said: "No, they shouldnt have any more power than appointed judges."Interviewer asked: What do you think of Red China?Respondent said: "It looks very nice on a white table cloth."3. ContextInterviewer asked: In challenging his opponent, the mayor demanded a dual meeting and debateon the issues...Respondent said: "Dueling ... in this day and age?"Asking questions correctly is both an art and a science. Whenever you design a questionnaire,pretest it. A pretest can help you isolate and avoid the ambiguity of language. Remember aquestionnaire is you operationalization of the concepts you will employ in the study. Your studywill be a good or as bad as your questionnaire...Good & bad in Questionnaire Design, Written By B2B InternationalA catalogue of the commonest errors and how to construct a questionnaire thatavoids themHOW OFTEN HAVE YOU STRUGGLED to respond to a questionnaire which just isnt quiteright for your answers? No doubt you became annoyed, frustrated and maybe even cynical aboutmarket research. These problems arise because of poor questionnaire design. The most frequenterrors in questionnaires are:
  • 19. 1. questions which dont quite mean what the researcher intended2. questions which dont probe to find out what the respondent really meant3. long and complicated questions4. questions which repeat what has already been asked5. questions which dont allow the respondent to answer in a way which is relevant6. questions which are inappropriate to the research method7. bad routing which leaves the interviewer wondering which question to ask next, or worse,routing to the wrong one8. too many questions9. poorly laid out questions which are badly grouped in the questionnaire10. Questions which have been missed out completely11. pages of the questionnaire which are missing or out of order.These errors can be eliminated or at least minimized if the researcher keeps two things clearly inmind: who will be answering the questions and what the research is aiming to achieve.The researcher must understand four fundamentals of questionnaire design:- What types of question can be asked- What types of response should be built in- How to lay out the questionnaire- How to test it.WHAT TYPES OF QUESTION?The researchers tool box contains three different types of question: behavioral, attitudinal andclassificatory.Answers to BEHAVIOURAL questions tell the researcher where a brand is now and where itcould be in the future. They show who is doing the buying and the pattern of purchasingamongst companies using the product. They allow the researcher to position a company orbrand in terms of its market share. Here are some examples of the subjects which behavioralquestions can address.Market size. Questions which need asking to assess the size of the market are: "Do you everbuy?", "How often do you buy?", "How much do you buy?" To make it easier for the respondenta time period may be built into the question: "Over the last month how much have you spenton... ?"Ownership patterns. Linked to market size questions are ownership questions. "Do you own acertain product?", "How long have you owned it?", "How many do you own?"Purchasing patterns. The researcher needs to understand the pattern of purchasing to work outthe best way to meet the demand. This understanding is acquired from questions such as "Howoften do you buy?", "When did you last buy?", "How much do you usually buy?"
  • 20. Behavioral patterns. Knowing the behaviors of a buyer helps the researcher provide the productat the right time at the right place. Questions which provide an understanding of purchasingbehavior include "How often do you buy/use/visit.. ? " and "Who is it at your company who hasthe final choice on the selection of a supplier?"Market shares. The shares which companies or brands hold can be assessed by asking buyers,"Which brands do you use?", "How much of this particular brand do you buy per month?"Future trends. Being able to predict or simply obtain a feel for future trends is one of the mostimportant yet most difficult of the researchers tasks. Pointers to the future can be obtained fromquestions such as "Do you intend to buy/visit/use... over the next six months?", "if yourpurchases this year were 100, what would you say they could be in five years time?", "What areyour reasons for saying that you will buy less in the future?"ATTITUDINAL questions establish respondents opinions and their image or perception of asubject. Their perceptions may, of course, be totally at variance with reality, but that may makethem all the more valuable as marketing data. Attitudinal research can help the researcher plana campaign to win sales by:showing why a company buys a certain brandshowing whether or not a demand is being satisfiedestablishing the views which people hold on certain products, services, prices, standards ofdelivery, etc.The third type of question in the researcher’s armoury is that which CLASSIFIES therespondent. Such questions allow answers from different companies to be grouped andcompared. Classification questions enable the researcher to segment the market and decidewhere marketing effort will be most rewarded. Common classifications used in industrial marketresearch are:size of company (measured by number of employees or turnover)the nature of the company’s business (usually determined by its Standard IndustrialClassification)geographical location.There may well be classifications which are special to a particular survey. Most industrialsurveys, for example, classify according to the size of the respondent company’s purchases, sothat the researcher can see the difference in attitudes or purchasing patterns between large andsmall companies.WHAT TYPES OF RESPONSE should be built in? Having selected the type of question(attitudinal, behavioural or classificatory), the researcher must decide on the response option.The questions can be either open ended or closed.Open ended questions leave the respondent free to give any answer, and verbatim responses are
  • 21. recorded. This type of question and answer is extremely flexible, as the researcher has notcommitted to a list of pre-coded responses.However, open ended questions are sometimes an easy option by researchers too lazy to think ofpossible pre-codes. Open enders should only be asked where the response is truly unknown.A closed question is one that has pre-coded answers. The simplest is the dichotomous questionto which the respondent must answer yes or no.Closed response questions save the respondent having to think of possible replies. They alsomake life easier for the interviewer who simply has to tick a box or circle a number. Moreover,they spare the coding staff difficult judgements which, if wrong, can skew the findings.A special type of pre-coded question is a scale. Scales can be of different types.Verbal rating scales. Typical of these is the five point scale running from very good through tovery poor. Verbal rating scales can be applied to almost any adjective - good/poor,happy/unhappy, satisfied/ dissatisfied, pleased/displeased, suitable/unsuitable, and so on.Instead of words, the researcher can use a numerical weighting scale. This usually runs up tofive or 10. The respondent needs to know the direction of the scale, viz:"I would like to ask your opinion on THE XYZ COMPANY. What score would you give it out of10 where one is totally dissatisfied and ten is totally satisfied?"Problems can occur if the researcher wants a rating on too many questions. Answers deteriorateas the questions grind on and the respondent repeats the same score in an effort to speed thecompletion of the interview.Diagrammatic rating scales. These, as the name suggests, ask the respondent to mark an answeron some continuum. This could be a line, a balance, or boxes. Diagrammatic scales are selfcompleted by the respondent and so they need to be clearly marked at each end of the scale.They can be more fun and faster to complete than the verbal or numerical scale.HOW TO LAY OUT THE QUESTIONNAIRE. The questions must now be organisedlogically and attractively. This is for the benefit of the interviewers, the respondents and thecoders. The layout depends very much on who will be administering the questionnaire, who willbe answering it, and the type of questions which are being asked.A well laid out questionnaire must have a sensible sequence of questions. The smooth flow of theinterview depends largely on the order in which questions are asked.Questions on image, purchasing behaviour and classification should all be arranged together.They should lead from one to another with the general subjects at the beginning and the moreparticular later on. This allows the respondent to relax and loosen up.For the same reasons, easy questions should precede the difficult ones. Sensitive questionsshould be placed towards the end, so that the respondent is "warmed up" and in autonomicresponse mode.
  • 22. The questions and responses should be clear and easy to read. If the boxes to tick (or numbers tocircle) are out of line or placed too far from the pre-coded responses, the interviewer could mis-cue an answer. There should be ample space for the interviewer to record verbatim responses.A clear and attractive layout is even more important in a self-completion questionnaire, where ithas a significant effect on the response rate.Good & bad in Questionnaire Design, Written By B2B InternationalA catalogue of the commonest errors and how to construct a questionnaire thatavoids themHOW OFTEN HAVE YOU STRUGGLED to respond to a questionnaire which just isnt quiteright for your answers? No doubt you became annoyed, frustrated and maybe even cynical aboutmarket research. These problems arise because of poor questionnaire design. The most frequenterrors in questionnaires are:12. questions which dont quite mean what the researcher intended13. questions which dont probe to find out what the respondent really meant14. long and complicated questions15. questions which repeat what has already been asked16. questions which dont allow the respondent to answer in a way which is relevant17. questions which are inappropriate to the research method18. bad routing which leaves the interviewer wondering which question to ask next, or worse,routing to the wrong one19. too many questions20. poorly laid out questions which are badly grouped in the questionnaire21. Questions which have been missed out completely22. pages of the questionnaire which are missing or out of order.These errors can be eliminated or at least minimized if the researcher keeps two things clearly inmind: who will be answering the questions and what the research is aiming to achieve.The researcher must understand four fundamentals of questionnaire design:- What types of question can be asked- What types of response should be built in- How to lay out the questionnaire- How to test it.WHAT TYPES OF QUESTION?The researchers tool box contains three different types of question: behavioral, attitudinal andclassificatory.Answers to BEHAVIOURAL questions tell the researcher where a brand is now and where itcould be in the future. They show who is doing the buying and the pattern of purchasingamongst companies using the product. They allow the researcher to position a company orbrand in terms of its market share. Here are some examples of the subjects which behavioralquestions can address.Market size. Questions which need asking to assess the size of the market are: "Do you everbuy?", "How often do you buy?", "How much do you buy?" To make it easier for the respondenta time period may be built into the question: "Over the last month how much have you spenton... ?"
  • 23. Ownership patterns. Linked to market size questions are ownership questions. "Do you own acertain product?", "How long have you owned it?", "How many do you own?"Purchasing patterns. The researcher needs to understand the pattern of purchasing to work outthe best way to meet the demand. This understanding is acquired from questions such as "Howoften do you buy?", "When did you last buy?", "How much do you usually buy?"Behavioral patterns. Knowing the behaviors of a buyer helps the researcher provide the productat the right time at the right place. Questions which provide an understanding of purchasingbehavior include "How often do you buy/use/visit.. ? " and "Who is it at your company who hasthe final choice on the selection of a supplier?"Market shares. The shares which companies or brands hold can be assessed by asking buyers,"Which brands do you use?", "How much of this particular brand do you buy per month?"Future trends. Being able to predict or simply obtain a feel for future trends is one of the mostimportant yet most difficult of the researchers tasks. Pointers to the future can be obtained fromquestions such as "Do you intend to buy/visit/use... over the next six months?", "if yourpurchases this year were 100, what would you say they could be in five years time?", "What areyour reasons for saying that you will buy less in the future?"ATTITUDINAL questions establish respondents opinions and their image or perception of asubject. Their perceptions may, of course, be totally at variance with reality, but that may makethem all the more valuable as marketing data. Attitudinal research can help the researcher plana campaign to win sales by:showing why a company buys a certain brandshowing whether or not a demand is being satisfiedestablishing the views which people hold on certain products, services, prices, standards ofdelivery, etc.The third type of question in the researcher’s armoury is that which CLASSIFIES therespondent. Such questions allow answers from different companies to be grouped andcompared. Classification questions enable the researcher to segment the market and decidewhere marketing effort will be most rewarded. Common classifications used in industrial marketresearch are:size of company (measured by number of employees or turnover)the nature of the company’s business (usually determined by its Standard IndustrialClassification)geographical location.There may well be classifications which are special to a particular survey. Most industrialsurveys, for example, classify according to the size of the respondent company’s purchases, sothat the researcher can see the difference in attitudes or purchasing patterns between large andsmall companies.WHAT TYPES OF RESPONSE should be built in? Having selected the type of question(attitudinal, behavioural or classificatory), the researcher must decide on the response option.The questions can be either open ended or closed.Open ended questions leave the respondent free to give any answer, and verbatim responses arerecorded. This type of question and answer is extremely flexible, as the researcher has notcommitted to a list of pre-coded responses.However, open ended questions are sometimes an easy option by researchers too lazy to think ofpossible pre-codes. Open enders should only be asked where the response is truly unknown.
  • 24. A closed question is one that has pre-coded answers. The simplest is the dichotomous questionto which the respondent must answer yes or no.Closed response questions save the respondent having to think of possible replies. They alsomake life easier for the interviewer who simply has to tick a box or circle a number. Moreover,they spare the coding staff difficult judgements which, if wrong, can skew the findings.A special type of pre-coded question is a scale. Scales can be of different types.Verbal rating scales. Typical of these is the five point scale running from very good through tovery poor. Verbal rating scales can be applied to almost any adjective - good/poor,happy/unhappy, satisfied/ dissatisfied, pleased/displeased, suitable/unsuitable, and so on.Instead of words, the researcher can use a numerical weighting scale. This usually runs up tofive or 10. The respondent needs to know the direction of the scale, viz:"I would like to ask your opinion on THE XYZ COMPANY. What score would you give it out of10 where one is totally dissatisfied and ten is totally satisfied?"Problems can occur if the researcher wants a rating on too many questions. Answers deteriorateas the questions grind on and the respondent repeats the same score in an effort to speed thecompletion of the interview.Diagrammatic rating scales. These, as the name suggests, ask the respondent to mark an answeron some continuum. This could be a line, a balance, or boxes. Diagrammatic scales are selfcompleted by the respondent and so they need to be clearly marked at each end of the scale.They can be more fun and faster to complete than the verbal or numerical scale.HOW TO LAY OUT THE QUESTIONNAIRE. The questions must now be organisedlogically and attractively. This is for the benefit of the interviewers, the respondents and thecoders. The layout depends very much on who will be administering the questionnaire, who willbe answering it, and the type of questions which are being asked.A well laid out questionnaire must have a sensible sequence of questions. The smooth flow of theinterview depends largely on the order in which questions are asked.Questions on image, purchasing behaviour and classification should all be arranged together.They should lead from one to another with the general subjects at the beginning and the moreparticular later on. This allows the respondent to relax and loosen up.For the same reasons, easy questions should precede the difficult ones. Sensitive questionsshould be placed towards the end, so that the respondent is "warmed up" and in autonomicresponse mode.The questions and responses should be clear and easy to read. If the boxes to tick (or numbers tocircle) are out of line or placed too far from the pre-coded responses, the interviewer could mis-cue an answer. There should be ample space for the interviewer to record verbatim responses.A clear and attractive layout is even more important in a self-completion questionnaire, where ithas a significant effect on the response rate.HOW TO TEST THE QUESTIONNAIRE. Questionnaires may have to be designed with onlya modicum of knowledge about the subject. In these circumstances mistakes are bound to occurunless the questionnaire is tested. The test should be carried out using the interviewing medium- ie telephone, face-to-face, postal – for which the questionnaire has been designed. The numberof interviews required to test a questionnaire could be as few as five to 10, though the more thebetter. Cost and time are invariably the constraints. In industrial market research, the projectmay use semi-structured questionnaires with a very small number of interviews. Piloting here isa continuous process, and modifications are made from the moment the study gets under way
  • 25. ConclusionInterpretation. When these guidelines are followed, the questionnaire becomes a powerful andeconomic evaluation tool.