2 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Constructing a Kinds of questionnaire information A questionnaire provides a tool for eliciting A questionnaire can help you obtain informa- information which you can tabulate and tion about what people do, what they have, discuss. In many evaluations, a questionnaire what they think, know, feel, or want. serves as the major source of information. Four different types of information may be dis- Writing questions and constructing a question- tinguished. Any one or a combination of these naire takes time and attention. Before you types may be included in a questionnaire. begin, it is essential to know what kind of evi- dence you need to fulfill the purpose of the KNOWLEDGE—What people study and to know how the information will be know; how well they understand used. something s Make a list of what you want to know. his type of question asks about what people What do you really want to find out? What do you want to achieve with the T know. Knowledge questions offer choices such as correct vs. incorrect or accurate vs. questionnaire? inaccurate. They may ask what respondents s Check to see if the information you need is believe is true or factual, or about awareness. already available somewhere else. For example: s DonÕt ask a question unless it has a use; that is, unless it relates to the purpose of What is the major cause of accidental deaths the study. Collecting too much information among children inside the home? adds to your time and expenses and can The most effective weight loss plan includes produce an information overload. exercise. Eliminate all the Ònice to knowÓ items that The ideal refrigerator temperature is ________. arenÕt really essential. Eliminate ambigu- ous questions as well. BELIEFS—ATTITUDES— s From the beginning, think through what OPINIONS you will do with each piece of information. hese terms refer to psychological statesÑthe What do you want to be able to say? Do you expect to use frequencies, percentages, T perceptions people hold, their thoughts, feelings, ideas, judgements, or ways of think- rankings, multivariate analysis, narrative ing. Questions may elicit respondentsÕ percep- remarks? tions of past, present or future reality, their s As you write questions, try to view them feelings about a subject, or their opinions. through your respondentsÕ eyes. For example: Will the question be seen as reasonable? Do you favor or oppose the reclassification of Will it infringe on the respondentÕs forestry land? privacy? In your opinion, does positive self-esteem Will the respondent be able and willing prevent drug abuse among adolescents? to answer the question? Do you think that lower prices would increase s Be selective and realistic. Know what beef consumption? information is needed, why, and how you What do you consider the biggest challenge plan to use it. facing our community in the next five years?
Q U E S T I O N N A I R E D E S I G N s s s 3BEHAVIOR—What people do Wording the uestions on behavior ask people what theyQ have done in the past, do now, or plan to doin the future. questions Wording the questions to obtain the intended information and to be understood by allFor example: respondents is a challenging task. When you Have you ever attended an Extension program write questions, consider three things: about nutrient crediting? 1) the particular people for whom the ques- Do you scout fields for pest problems? tionnaire is being designed; How are you currently using the information 2) the particular purpose of the question- gained in the Healthy Eating workshop? naire; and 3) how questions will be placed in relation toATTRIBUTES—What people are; each other in the questionnaire.what people have Some suggestions appear below. ttributes are a personÕs personal or demo-A graphic characteristicsÑage, education,occupation, or income. Attribute questions ask (adapted from Sawer, 1984; Dillman, 1978; Newsome, n.d.)people about who they are, rather than what s Use simple wording. Adapt wording tothey do. the vocabulary and reading skills of your respondents but donÕt talk down to them.For example: Are any words confusing? Do any words Where do you currently live? have double meanings? How many children do you have? s Avoid the use of abbreviations, jargon, What percentage of your household income or foreign phrases. We use a lot of jargon comes from off-farm employment? in Extension. Will the respondents under- stand terms such as ÒCNRED,Ó ÒlearningTo write meaningful questions, be clear about experiences,Ó Òlife skills,Ó or Òfocusthe objectives and type of information groupsÓ?desiredÑwhether it is information about s Be specific. A question about older youthknowledge, attitudes/beliefs/opinions, behav- should specify what age or grade is con-ior, or attributes. Otherwise, the questionnaire sidered Òolder.Ó Likewise, in the questionmay elicit opinions when the actual intent is to ÒHow many times did your 4-H club meetdocument behavior. last year?Ó specify whether Òlast yearÓLikewise, questions related to each type of refers to 1994, 1994Ð1995, the last 12information present different writing prob- months, or September 1994ÐAugust 1995.lems. Questions concerning attitudes tend to s Use clear wording. Words such as Òregu-be more difficult to write given the complexity larlyÓ and ÒoccasionallyÓ mean differentunderlying most attitudes. Pay careful atten- things to different people. Some vaguetion to wording. In contrast, questions about terms include: majority (more than half ofknowledge, behaviors and attributes are more what?); often (daily? twice weekly?straightforward. weekly?); government (state? federal? local?); older people (how old?).Remember, the response or information youobtain is only as good as the question. If youdonÕt get the type of information you want, itis probably because you didnÕt ask the rightquestion!
4 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N s Include all necessary information. In s Avoid making assumptions. Questions some cases, respondents may not know such as ÒHow many children do you enough to adequately answer a question. have?Ó or ÒDo you prepare beef when you For example: ÒDo you agree or disagree invite friends over to eat?Ó make assump- with the proposed plan to expand the role tions about the respondentsÑthat they of 4-H volunteers in our community?Ó have children and invite friends over to Respondents may not know what the plan eat. A better set of questions would start is. Provide a statement summarizing the with the first question establishing the sit- plan. uation, followed by the question of inter- s Avoid questions that may be too est. For example: ÒDo you have children?Ó precise. PeopleÕs lives are usually not so ÒHow many children do you have?Ó orderly that they can recall exactly how s Avoid bias in questions. Biased questions many times they ate out last year or how influence people to respond in a way that many Extension meetings they attended in does not accurately reflect their positions. 1995. To help respondents formulate an A question can be biased in several ways: answer, the response category might (1) when it implies that the respondent provide a range to select from, for should be engaged in a particular behav- example, 0Ð5, 6Ð10, 11Ð15, etc. ior; (2) when the response categories are s Phrase personal or potentially incrimi- unequal or loaded in one direction; nating questions in less objectionable (3) when words with strong positive or ways. Being asked to indicate drug use, negative emotional appeal are used, such income level, or ethnic background may as Òfreedom,Ó Òequality,Ó Òboss,Ó be objectionable to some respondents. One Òbureaucratic,Óetc. method is to ask respondents to select Here are some examples of biased questions: from among broad categories (income less 1. More farmers in Saymore County are using than $10,000, $10,000Ð$20,000, $20,000 and Superb than any other variety of alfalfa. Do over, etc.) rather than specifying precise you use Superb? information. A series of questions may also be used to soften or overcome the 1. No objectionable nature of certain informa- 2. Yes tion. This question implies the respondent s Avoid questions that are too demanding should be using Superb. and time consuming. Examples of such 2. How would you rate the housing in which questions are, ÒPlease rank the following you live? 15 items in order of their importance to 1. Satisfactory youÓ or ÒIn 25 words or less, what is your philosophy of 4-H?Ó 2. Good s Use mutually exclusive categories. 3. Excellent Make sure that only one answer is possi- No negative options provided. ble. In the example: ÒHow did you hear 3. Do you agree that funding for Extension in about the Extension seminar?Ó the your county should be increased? response categories are: Òfrom a friend, from a relative, from the newspaper, at 1. No work, from the county office, at an 2. Yes Extension meeting.Ó The respondent may have heard about the Extension seminar from a friend at work, for example, so that more than one answer is possible.
Q U E S T I O N N A I R E D E S I G N s s s 5 This is a leading question. A better ques- Better tion would state: 1. 0 acres Do you agree or disagree that Extension 2. 1–49 acres funding should be increased? (Circle one.) 3. 50–99 acres 1. Strongly disagree 4. 100–149 acres 2. Disagree 5. 150–199 acres 3. Agree 6. 200–249 acres 4. Strongly agree 7 250 acres and overs Avoid double-barreled questions. ÒDid s Use complete sentences. Trying to keep the poultry production seminar help you questions simple and concise may result in to identify ways to improve the sanitation questions that are cryptic and easily mis- and increase the nutrition of your cage understood. bird operation?Ó ItÕs better to ask about s Plan ahead. Identify each question and ÒsanitationÓ and ÒnutritionÓ separately. each response item with a number or letter Other questions may be too ambiguous; for easy tabulation. for example: ÒDo you favor legalization of marijuana for use in private homes but not Types of questions in public places?Ó This gives no opportu- Questions can be open- or close-ended. The nity for people to respond in favor of both following is adapted from Sawer, 1984. places, to oppose both places, or to oppose home but favor public use. Open-ended questions allow respondents tos Make the response categories clear and provide their own answers. This gives them logical. Too often the answers are confus- the opportunity to express their own thoughts, ing, not in logical order or spaced so that but also requires more effort in terms of their numbers or figures are hard to interpret. responses. Open-ended questions tend to For example: produce varieties of answers and are more dif- ficult to analyze. Poor spacing 1.0 acres Close-ended questions list answers, and respondents select either one or multiple 2.1–9 acres responses. These questions produce more 3.10–99 acres uniform answers than open-ended questions, 4.100–499 acres but depend upon your knowing and including 5.500–999 acres all relevant responses in the list. Responses to 6.1,000 acres close-ended questions must be exhaustive and also mutually exclusive in providing for the Poor logic selection of a single response. 1. 1,000 acres Examples of open- and close-ended questions 2. 999–500 acres are explained further on the next several 3. 499–100 acres pages. 4. 99–10 acres 5. 9–1 acres 6. 0 acres
6 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Open-ended questions n open-ended question is often the easiest A question on one topic structured in different ways A way to ask for information, but the responses are not easy to analyze. Answers are 1. Open-ended likely to be varied so you will need to catego- What would you like to see as the main program rize and summarize them. Think about how emphasis next year? you will analyze the narrative data. Open-ended responses can be used to: 2. Close-ended with ordered responses How important to you are each of the following s Stimulate free thought, solicit creative sug- possible program emphases? (Circle one for each gestions, or recall information learned. item.) s Probe for more detail. None Little Some Much Examples: A. Effective parenting 1 2 3 4 1. What do you think should be done to improve B. Child development 1 2 3 4 the 4-H program in this county? C. Guidance & discipline1 2 3 4 2. Name the five basic food groups. D. Communications 1 2 3 4 3. Please indicate how you intend to use the infor- mation obtained during the workshop. 3. Close-ended with unordered response 4. We are interested in any other comments you choices might have concerning your role as a volunteer Which of these four topics would you most like to leader. Please write in the space below any see as the primary program emphasis next year? thoughts youÕd like to share with us. (Circle number of your answer.) When asking for a numeric response, include 1 Effective parenting the unit of measurement to be used. 2 Child development 5. Please list the number of acres (if any) of tem- 3 Guidance and discipline porary pasture you planted in 1988. 4 Communication A. ____ Acres of wheat 4. Partially close-ended B. ____ Acres of oats What topic do you feel should be the main program C. ____ Acres of rye grass emphasis for next year? (Circle number of your D. ____ Acres of clover answer.) E. ____ Acres of summer annuals 1 Effective parenting F. ____ Other, please specify 2 Child development Open-ended questions are also appropriate 3 Guidance and discipline when respondents are asked to supply a spe- 4 Communication cific answer and a large number of responses 5 Other (please specify) are possible (see example 3 above) or when all ___________________ the possible answers are not known. They are often used at the end of a questionnaire to ask (Dillman, 1978) respondents for additional comments (example 4 above).
Q U E S T I O N N A I R E D E S I G N s s s 7Close-ended questions Example: here are a variety of ways to write close- 1. What does the word ÒnutritionÓ mean to you?T ended questions. Some require answers thatfall along an implied continuum (as in rating (Circle one number.) 1 Getting enough vitaminsscales); others supply answers in no particular 2 The food you eat and how your bodyorder (lists). Some questions employ multiple uses itchoice options (Òcheck all that applyÓ); others 3 Having to eat foods I don’t likeprovide relevant answers but allow respon-dents to add others not in the list. The follow- 4 Having good healthing section gives examples of close-ended Rating scalequestions. Often, respondents are asked to indicate their choice at the most appropriate point on a scale.Two-option responses Whether you use a scale of three, four, five orThis is the simplest response format. Options more categories depends on the question, themay include: NoÐYes, DisagreeÐAgree, amount of differentiation that is possible andFalseÐTrue, OpposeÐFavor. desirable, and the respondentsÕ capacity to Example: answer. 1. Do you remove the clippings from your lawn Examples: after mowing? 1. To what extent do you agree or disagree with 1 No the new zoning code? (Circle one.) 2 Yes 1 Strongly disagreeDepending upon the information you desire, 2 Mildly disagreethis may be the most appropriate format. It is 3 Neither agree or disagreeoften used as the first question in a series ofquestions on one topic. However, using a 4 Mildly agreerating scale or a ranking (when appropriate) 5 Strongly agreeelicits more information. 2. When purchasing new herd bulls, how impor-Starting with either positive or negative tant are the following traits in your selectionresponse options appears to have little effect process? (Circle one number for each selectionon response. Neither does it matter whether trait.)Yes or No is listed first. But you do need to be Selection Of little Highlyconsistent in the order you follow throughout trait importance importantthe questionnaire. A. Performance 1 2 3 4 5 6 7One best answer B. Conformation 1 2 3 4 5 6 7These questions can be used to solicit informa-tion or to test knowledge. They are appropriate C. Pedigree 1 2 3 4 5 6 7when all relevant choices are known and D. Breed 1 2 3 4 5 6 7listed. Respondents are provided with a list ofanswers and asked to check or circle the choice For greater differentiation, use a numerical scalethey feel is the best. Responses are indepen- from 0 or 1 to some number (see the seconddent of each other, rather than gradations example above). A five-point option seriesalong a continuum. seems to be best for measuring attitudes; a four-point option series appears useful for ratings (excellent, good, fair, poor). Some people may relate best to a 10-point scale.
8 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Another decision youÕll need to make is Examples: whether to use an even or odd number of 1. Within your 4-H club, describe the extent to response options. An odd number of categories which you were included in making important provides a middle or neutral position for the decisions. (Circle one number.) respondent to select, while an even number 1 Never forces the respondent to take sides. This is 2 Rarely appropriate when you want to know in what direction the people in the middle are leaning. 3 Sometimes Below are some examples of rating response 4 Often categories. 2. How do you feel about this statement, ÒI wish this community had more outdoor recreation Very dissatisfied No help at all Somewhat dissatisfied Slightly helpful centers?Ó (Circle one number.) Somewhat satisfied Fairly helpful 1 Strongly disagree Very satisfied Very helpful 2 Mildly disagree Strongly unfavorable Strongly disagree 3 Neither agree nor disagree Generally unfavorable Disagree somewhat 4 Mildly agree Uncertain Uncertain Generally favorable Agree somewhat 5 Strongly agree Strongly favorable Strongly agree Responses must reflect a clear difference and be balanced both positively and negatively. A Decreased Poor poor set of responses would: Òdecreased, Stayed the same Fair stayed the same, increased a little, increased Increased Good Excellent somewhat, increased a lotÓ (only one negative and three positive choices are given). Five categories are about the most you should It is not necessary to use the same response use when listing words in the responses. categories for each question, but do it where Remember to keep the positive and negative possible. Most importantly, choose responses options balanced. that are appropriate to the question. Ordered choice “Other, please specify” In this type of question, the responses are Here the respondent is offered a choice of an usually intended to measure degree or intensity answer plus the opportunity to enter his/her in an ordered sequence or scale. Ordered choice own answer under Òother, please specify.Ó This questions are particularly suited for evaluating protects you against leaving out an important attitudes. They are appropriate when the topic answer choice. It also means that you will have is well-defined and the choice represents a gra- narrative text to analyze. dation along a single dimension. Think about what you will do with these responses. Too often this type of information is never used because it cannot be added up easily.
Q U E S T I O N N A I R E D E S I G N s s s 9 Examples: Paired comparisons 1. What do you consider the main responsibility Respondents are asked to compare one item to of your county 4-H agent? (Circle one another, usually expressed in terms of number.) Òeither/orÓ or one item ÒversusÓ another. 1 Work with people who request help Example: 2 Work with 4-H members 1. In comparing beef to other meats, which does 3 Work with volunteer 4-H leaders your family use more often? (Choose one from 4 Plan and organize county youth events each comparison and circle the number.) 5 Organize and expand new 4-H clubs 1 Beef OR 2 Poultry 6 Other, please specify _______ 3 Beef OR 4 Lamb 5 Beef OR 6 Pork 2. Which of these community recreational facili- ties do you most frequently use? (Circle one 7 Beef OR 8 Wild game number.) (venison, etc.) 1 Parks Matching 2 Tennis courts Respondents are asked to match responses to a list of items. 3 Swimming pools 4 Other ______ Example: 1. Match each food to the proper food group byItems in a series putting the correct lower case letter from theWhen several questions use the same response right side in the blank.category, it is possible to present the responses A. __ Wheat roll a. Meat and meat productsin a table, rather than write separate questionsfor each. B. __ Nectarine b. Milk and milk products C. __ Ham c. Fruits and vegetables Example: D. __ Yogurt d. Breads and cereals 1. How often do you eat the following meats? (Circle one number for each meat.) E. __ Pumpkin e. Sweets F. __ Oatmeal Once/ 1–3 times 4–6 times Never week week week DailyA. Beef 1 2 3 4 5B. Lamb 1 2 3 4 5C. Pork 1 2 3 4 5D. Poultry 1 2 3 4 5E. Fish 1 2 3 4 5
10 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Close-ended questions with multiple Lists choice answers: Check all that apply A list provides a series of answers. Respon- This common response format is actually a dents may choose one or more depending on series of ÒyesÓ or ÒnoÓ items. It is a fast and the instructions. easy way to obtain such information and also save space. DonÕt make the list too long or the Examples respondents may not consider each item. 1. Listed below are some adjectives which might be used to describe a person. Please indicate for Examples: each adjective, whether the adjective does or 1. What steps have you taken to set up a busi- does not describe you. (Circle one number for ness? (Check all that apply.) each adjective.) s a. Improved product or skills Describes Does not DonÕt s b. Defined product or service me describe me know s c. Identified customers a. Ambitious 1 2 3 s d. Researched market potential b. Happy 1 2 3 s e. Filed business name c. Idealistic 1 2 3 s f. Established recordkeeping system d. Outgoing 1 2 3 s g. Applied for resale tax number 2. From the list provided, select THREE adjec- s f. Other tives which best describe you. ________________________________ (Place the letter of the adjective on the lines provided.) 2. What information would you like covered in 1. _____ a. Ambitious the next Extension workshop? (Check all choices.) 2. _____ b. Happy s a. Container production 3. _____ c. Idealistic s b. Landscape design d. Outgoing s c. Disease control s d. Nursery layout s e. Weed control practices s f. Greenhouse management practices
Q U E S T I O N N A I R E D E S I G N s s s 11RankingRank ordering is a multiple-choice option. Formatting theRespondents are given various responses and questionnaireasked to rank them in order of importance or After you have selected your questions, youÕllindicate a Òtop three.Ó need to make a series of decisions about the questionnaire formatÑits appearance, length, Examples: and the order in which the questions will 1. What would you like to know more about? appear. The questionnaire should be pleasing Select three responses from the list and rank to the eye and easy to complete. them in order of 1, 2, 3. The following guidelines offer some tips to 1. ___ a. What to eat to look better help you put the questionnaire together. 2. ___ b. How food affects you s Begin with an introduction that includes 3. ___ c. Weight control the questionnaireÕs purpose, identifies its d. Health foods source, explains how the information e. Physical conditioning obtained will be used, and assures respon- through diet dents of confidentiality. In mailed ques- f. Vitamins tionnaires, reinforce the points you made in the cover letter. 2. What would you like to know more about? Select three responses from the right hand s The first questions should be easy, avoid- column and rank them in order of first, second, ing controversial topics. Write interesting and third choice. questions that are clearly related to the questionnaireÕs purpose. DonÕt use open- 1. __ 1st choice a. What to eat to ended or long questions with lengthy look better answer choices in the beginning of the 2. __ 2nd choice b. How food affects questionnaire. you s Address important topics early, rather than 3. __ 3rd choice c. Weight control late, in the questionnaire. d. Health foods s Arrange questions so that they flow natu- e. Physical conditioning rally. Keep questions on one subject grouped together. Start with general ques- f. Vitamins tions and then move to those that are specific. s Try to use the same type of question and response throughout a series of questions on a particular topic. For example, donÕt needlessly break a respondentÕs concentra- tion by using a multiple choice format fol- lowed by a yes/no question, followed by an open-ended question. s Place demographic questions (age, sex, income level, etc.) at the end of the ques- tionnaire. s Print it in an easy-to-read typeface. s A numbered response should mean the same thing throughout the questionnaire.
12 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Example: s Pre-code as many items and response cate- If you begin with: gories as possible to help tabulate and analyze data more quickly. When data is 1 No precoded, it can be entered directly from 2 Yes the questionnaire. Try to position the donÕt switch to: response blanks in the same place on the 1 Yes page to make tabulation easier. 2 No s Use transitional statements to enhance continuity. Transitional statements serve s Avoid making respondents turn a page in three functions: 1) to signal that a new the middle of a question or between a topic is about to begin; 2) to start new question and answer. pages; and 3) to break up the monotony of s Be sure that the question is distinguishable a long series of questions. from the instructions and the answers. You Examples: could put the instructions in boldface or italics. Dillman (1978) suggests using Next, we would like to ask you several lower case letters for questions and upper questions about the organizations you case letters for answers. belong to in your community. s Questions and answers are easiest to read Another important purpose of this survey if they flow vertically. By placing answer is to learn how you feel about the work of choices under questions (rather than side service organizations in your community. by side), the respondent moves easily Finally, we would like to ask a few ques- down the page. If you feel this format tions about you to help us interpret the results in too much wasted space, you may results. wish to reorganize your questions. s Make sure that the respondent is referring Example: to the same program mentioned in the 1 Excellent questionnaire and defining it similarly. A Òvalidation itemÓ (Bennett, 1982) at the 2 Good beginning of the questionnaire identifies 3 Fair the program and sets the stage for the 4 Poor questions to follow. It is a brief summary of the programÕs activities and the people Rather than: __ Excellent __ Good __ Fair involved. __ Poor Example: s Give directions about how to answer. Include directions in parentheses immedi- The Dell County Extension family living ately following questions. It is better to program included a variety of activities repeat directions too often than not during 1989 focusing on teaching money enough. Here are some examples of spe- management and budgeting skills to help cific instructions you might use: Circle the families manage their resources better. number of your choice; circle only one; check These activities included lunch and learn all that apply; please fill in the blank; enter programs, computer budgeting workshops, whole numbers; please do not use decimals or letter series and short courses. Consumers fractions; etc. from across the county attended these activities on Money Management Skills.
Q U E S T I O N N A I R E D E S I G N s s s 13Filter or screen questionsSome questions may not apply to all respondents. For these Òscreen questions,Ó make it clearwho should answer the question. Also be sure to give directions for those not expected torespond. Dillman (1978) makes three suggestions: 1) use arrows to guide respondents from one question to the next; 2) indent all questions that may be screened; or 3) use boxes to direct respondents past the questions(s) they donÕt need to answer. Examples: Q-5 Do you own or rent the home in which you now live? 1 Own home If you own your home, 2 Rent home skip from here to Q-14 on the next page (If you rent) Q-6 How much is your monthly rent? 1 Less than $100 2 $100 to $199 3 $200 to $299 4 $300 or more OR Q-5 Do you own or rent the home in which you now live? (Circle the number of your answer.) 1 Own home 2 Rent home (if you own your home) (if you rent your home) Q-6a How much is your monthly house Q-6b How much is your monthly rent? payment (without property taxes)? 1 Less than $100 1 Less than $100 2 $100 to $199 2 $100 to $199 3 $200 to $299 3 $200 to $299 4 $300 or more 4 $300 or more Q-7a How much per month do you Q-7b Which, if any, of these is pay for electricity, garbage included in your monthly rent? collection, heat and water? (Circle all that are included.) 1 Less than $25 1 Electricity 2 $25 to $74 2 Garbage 3 $75 to $124 3 Heat 4 $125 or more 4 None of the above
14 s s s P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N Pretesting the s Ask colleagues to review the question- naire critically. Let coworkers read the questionnaire questions to see if the wording and Pretesting is an indispensable part of question- instructions are clear, and if the question- naire design. Many practitioners feel that if the naire will accomplish the study objectives. resources to pretest the questionnaire are not Consider reviewersÕ comments carefully; available, itÕs best to postpone the study until then decide if they enhance the question- the resources are available. naire. s Select people as similar to your respon- This means you must examine individual dents as possible to pretest the ques- questions, as well as the whole questionnaire, tionnaire. Choose people to represent a very carefully. Allow enough time to incorpo- cross-section of the population. rate any revisions. Unfortunately, too many people consider pilot testing a perfunctory task s Simulate the actual data collection pro- if they consider it at all. cedure as closely as you can,whether it is a mail survey, telephone or direct inter- According to Salant and Dillman (1994), any view. If youÕre using a mail questionnaire, pretest needs to answer the following ques- have people answer it without any help tions: and afterwards, ask for their suggestions. Ñ Does each question measure what it is In an interview, have the interviewer intended to measure? conduct the pilot test, either by phone or face-to-face, as it will actually be done. Ñ Do respondents understand all the words? s Obtain feedback about the form and Ñ Are questions interpreted similarly by all content of the questionnaire. Were any respondents? questions misunderstood? Were the direc- Ñ Does each close-ended question have an tions clear? Was the questionnaire too long answer that applies to each respondent? or too difficult? How long did it take to fill Ñ Does the questionnaire create a positive out? Was there enough space for impressionÑone that motivates people to responses, etc.? answer it? s Assess whether the questions produce Ñ Are the answers respondents can choose the information needed to satisfy the from correct? (Are some responses studyÕs purpose. missing? Do some questions elicit uninter- s Try the tabulation and analysis proce- pretable answers?) dures to make sure that the questionnaire Ñ Does any aspect of the questionnaire yields data that can be analyzed in the way suggest bias on the part of the researcher? that is needed. s Revise. Check the final draft. Go over each question and ask: What will the informa- tion obtained from the question mean? How much will it contribute to the study? (Adapted from Rohs, 1985; Sudman and Bradburn, 1986; Salant and Dillman, 1994)
Q U E S T I O N N A I R E D E S I G N s s s 15ReferencesBabbie, Earl R. Survey Research Methods. Belmont, CA: Wadsworth Publishing Co., 1973.Bennett, Claude F. Reflective Appraisal of Programs (RAP): An Approach to Studying Clientele-Perceived Results of Cooperative Extension Programs. Ithaca, NY: Cornell University Media Services, 1982.Dillman, Donald A. Mail and Telephone Surveys: The Total Design Method. New York: John Wiley and Sons, Inc., 1978.Fischer, Christy and Jeffrey D. Layman. Edge: Constructing a Questionnaire. Ohio State University: Ohio Cooperative Extension Service, 1986.Newsome, Bob W., et al. Northeast Area Evaluation Process. Kansas State University, Kansas State Cooperative Extension Service, n.d.Rohs, F. Richard. Questionnaire Construction. Athens, GA: Cooperative Extension Service, 1985.Salant, Priscilla and Don Dillman. How to Conduct Your Own Survey. New York: John Wiley & Sons, Inc. 1994.Sawer, Barbara J. Evaluating for Accountability. Corvallis: Oregon State University Extension Service, 1984.Sudman, Seymour and Norman M. Bradburn. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass Publishers, 1982.
AppendixAttribute information s Phrase income questions carefully. Do hen you collect attribute data (personal you want to know the householdÕs (orW characteristics such as age, education,income,) it is often a good idea to formulate respondentÕs) annual income? Monthly income? Net? Gross? Taxable? Take-home pay? Some people are sensitive to incomeresponses you can compare to existing data; questions. Instead of asking ÒWhat wasfor example, categories from a previous study, your total household income in 1995?Ó youor the same categories as reported in U.S. could ask, ÒWhich of the following cate-Census data. You can then compare your data gories best describes your total householdto the census to demonstrate how closely your income for 1995?Ó Offer a choice of severalsample reflects the general population. broad dollar amounts.Even if you have no comparisons currently in s Use appropriate employmentmind, collecting this type of information about categories. Employment status is onerespondents can contribute to a Òdata bank.Ó It attribute that is hard to describe. The fol-will allow you to build a Òparticipant lowing example is a fairly lengthy one.profileÓthat describes participants in terms of (The respondent is asked to read the entirekey variables, compares the characteristics of list, then select the one response whichthose who participate and those who donÕt, best describes his/her employment status.)assesses the degree to which target audiences A. Employed full-time in the work forceare being reached, and plan strategies for pro- B. Employed part-time in the work forcemoting future programs effectively. C. Unpaid full-time in farm, family or homeHere are a few tips for collecting such data. business(Adapted from Sawer, 1984) D. Unpaid part-time in farm, family or home business s Place attribute questions at the end of E. Unemployed, seeking work the questionnaire. F. Unemployed, not seeking work s Avoid overlapping categories. Use exclu- sive categories; for example 25Ð34, 35Ð44, G. Student, employed part-time 45Ð54, etc., not 25Ð35, 35Ð45, 45Ð55. H. Student, not seeking work s Ask age questions directly, but be diplo- I. Retired, employed part-time matic. Some respondents may perceive J. Retired, not seeking work ÒHow old are you?Ó as a little blunt. A simpler breakdown is: ÒWhat is your age?Ósoftens the request. ÒPlease circle the number of the category A. Employed which includes your ageÓ is inoffensive. If B. Unemployed, seeking work you need more precisely refined figures, C. Not in the labor force you could ask ÒWhat was your age at your most recent birthday?Ó(Most people hope they havenÕt yet celebrated their ÒlastÓ birthday.) Also avoid asking for a month and/or year of birth unless itÕs absolutely necessary, since such information will require many extra calculations.
Q U E S T I O N N A I R E D E S I G N s s s 17Some examples of attribute questions follow. 7. What is your race? (Circle one number.) 1. What is your age? (Circle one number.) 1. Anglo or white 1. Under 18 2. Black 2. 18–24 years 3. Hispanic 3. 25–34 years 4. Asian 4. 35–44 years 5. Native American 5 45–54 years 6. Other ____________ 6. 55–64 years 8. Is your current agricultural operation a single-family operation, a partnership, a 7. 65 or older family-held corporation, or a nonfamily- 2. Are you married? held corporation? (Circle one number.) 1. No 1. Single-family operation 2. Yes 2. Partnership 3. How many people live in your household, 3. Family-held corporation counting yourself? 4. Non-family-held corporation _____________________ 5. Other (specify)____________ 4. Are you employed outside the home? 9. What was the approximate gross value of 1. No farm sales from your operation last year 2. Yes If you work outside the including crops sold, animals sold, and home, do you work: dairy products? (Circle one number.) 1 Full-time? 1. Less than $2,500 2 Part-time? 2. $2,500 to $4,999 5. What is the highest level of education that 3. $5,000 to $9,999 you have completed? (Circle one number.) 4. $10,000 to $19,999 1. Some grade school (1–8) 5. $20,000 to $39,999 2. Some high school (grades 9–12) 6. $40,000 to $99,999 3. High school graduate 7. $100,000 to $199,999 4. Some college or technical school 8. $200,000 to $499,999 5. College graduate or beyond 9. $500,000 or more 6. Where do you live? (Circle one number.) 10. In the past year did you work off-the-farm 1. Rural farm in a part-time or full-time job? (Circle one 2. Rural community (less than 2,500 number.) people) 1. Did not work off the farm 3. Town between 2500–25,000 people 2. Was employed part-time off the farm 4. City between 25,000–50,000 people 3. Was employed full-time off the farm 5. City over 50,000 people