FAIRFAX COUNTY DEPARTMENT OF SYSTEMS MANAGEMENT FOR HUMAN SERVICES April 2003 Informational Brochure Survey Questionnaire Design Table of Contents A survey is a means of gathering information about a particular population Preliminary 1 by sampling some of its members, usually through a system of standardized Considerations questions. Surveys can be conducted by mail, telephone, personal Survey Modes or Types 2 interview, or Internet. They can be administered either to individuals or groups. The primary purpose of a survey is to elicit information which, after Survey Questionnaire 4 Design evaluation, results in a profile or statistical characterization of the Question Content 5 population sampled. Questions may be related to behaviors, beliefs, attitudes, and/or characteristics of those who are surveyed. Question Formats 6 Pretest 7 Preliminary Considerations Glossary 7 Need for a Survey—Since almost all surveys can be costly, it is critical to discern whether or not the study needs to be done. Begin by contacting persons knowledgeable in the field and by performing an Types of Questions: environmental scan of other studies• Attitudes—Respondents’ conducted on the topics of interest. views, perceptions, or This work should provide the 4. Can the desired information feelings. How they feel (usually judgmental). answers to the following questions: actually be collected by a survey or would another form of• Beliefs—What 1. Have studies of this subject been respondents think is research be more appropriate? true; their perception of done previously? reality (assessment 5. Is there adequate time and oriented, taps what they 2. Is there literature enough on the resources available to conduct a know). subject to answer the question survey without skipping steps in• Behaviors—What (i.e., books, periodicals, respondents do the process? reports)? (present, past, and future). Once the need for a survey is 3. Have other county organizations determined, a problem statement• Attributes—Personal or investigated this area, and do demographic and objective must be developed for they have information available characteristics (age, the survey. income, occupation). on the subject?
Page 2 Survey Questionnaire DesignProblem Statement—A clear, be located? early in the planning processconcise statement of the so that expectations for what 4. Does the informationproblem to be studied and/or the survey can accomplish collected need to bethe information desired should remain realistic in light of statistically valid and doesbe put into writing. It is helpful financial constraints. it need to be generalized toto list possible causes of the a larger population? Survey Mode—The next step inproblem, as well as possible the process is to determinesolutions. This will help clarify 5. What kinds of analyses which survey mode to use. Thethe survey objectives. would be useful for survey mode is the type of understanding the surveySurvey Objectives—Survey survey that will be conducted. results?objectives will be concerned The most frequently usedwith the following issues: 6. Will the statistics resulting modes include face-to-face or from the analysis of the personal interviews, telephone1. What information is needed survey data be appropriate interviews, and written in order to understand the for the sampling plan used interviews which are usually problem, its causes, and as well as the questions to conducted by mail or Internet. possible solutions? be answered? The factors that will determine2. How will the information be which mode to choose include Survey Budget—When used and by whom? financial constraints; resource conducting a survey, an3. What/who is the population adequate budget must be constraints; and question to be studied and can all developed to cover all phases length, complexity, and members of the population of work. This should be done sensitivity.Survey Modes or TypesA. Face-to-Face InterviewFace-to-face interviews or personal interviewsare surveys conducted in person by aninterviewer who usually travels to the personbeing surveyed.• Pros—High response rates; can clarify questions, if necessary; control over respondent selection; can use longer, more complex questionnaire; and easier to and training interviewers, contacting motivate the respondent. respondents, travel arrangements). Also,• Cons—High costs, time-consuming, and more there is a tendency for respondents to give administrative requirements (i.e., selecting socially acceptable answers.
Page 3B. Telephone Interviews influenced, easy to administer and relativelyTelephone interviews are usually conducted low costs, can cover a wide geographicalfrom a central office that places telephone calls area, and more manageable for handlingto selected households or businesses. large samples.• Pros—Good response rates, fast, some • Cons—Questionnaire may be given to anonymity for respondents in answering someone else to fill out or may not reach the questions, and control over respondent desired respondent; most difficult type of selection. If a comprehensive list of the questionnaire to design; hard to interpret target population is open-ended available, the likelihood questions; cannot of obtaining a control sequence in representative sample which respondents is high. answer questions; and time-• Cons—Questions must consuming, given be short and not periodic mail-out complex; cannot requirements. control interruption by others in household/ D. Internet Questionnaire office; hard to find An Internet questionnaire is a form of a written persons at home, and those survey. Respondents may be invited to that are at home may resent participate in the survey through email or intrusion; there is mounting because they visit a particular web page. displeasure among households receiving • Pros—Fast to conduct and tabulate, some unsolicited telephone calls; requires training software products allow questionnaires to be and quality control monitoring of the customized depending on the respondent’s interviewers; and is usually difficult to target answers, avoids interviewer bias and a specific geographical location. distortion, answers unlikely to be sociallyC. Mail Questionnaires influenced, easy to administer, andMail questionnaires are written surveys that are relatively low costs.sent through the mail to selected members of • Cons—Information transferred via thethe population to be surveyed. Internet may not be confidential; poor control over respondent selection; follow-up• Pros—Good response rates with rigorous difficult to conduct; difficult to obtain follow-up procedures, relatively easy to probability sample; and, like mail surveys, obtain a listed population and locate this is the most difficult type of questionnaire respondents, can avoid interviewer bias and to design. distortion, answers unlikely to be socially
Page 4 Survey Questionnaire DesignSurvey Questionnaire DesignThe survey questionnaire should be designed toinclude elements which make the survey B. General Layoutpertinent and relevant to the population to be The layout of a written questionnaire can have assampled, thereby maximizing response rates much to do with response and error rates as doand minimizing error or bias. the actual questions. Therefore, the followingA. Components factors need to be carefully addressed:The following sections 1. Length—All surveys should be as briefnormally comprise a as possible. Mail and telephone surveysquestionnaire: should be no longer than 10 to 151. Request for minutes. Personal interviews should not Cooperation—This extend beyond 30 minutes. might be a brief 2. Appearance—Mail and Internet introductory paragraph surveys should give the appearance that (or speech) at the they will be easy to complete. Neat, beginning or could be a orderly written questionnaires with a lot comprehensive cover of white space will increase response rates. letter. It should highlight the reason for the 3. Instructions—Clear, unambiguous, and easily survey, voluntary participation, readable instructions work best. In mail and confidentiality, and willingness to provide a Internet surveys, it helps to offset instructions copy of results to respondents if desired. from the rest of the text.2. Instructions—Always simple, clear, and 4. Vertical Flow—Logical question and section repetitive where necessary. Keep to a sequencing is critical. Avoid jumping from minimum and make sure they are easy to topic to topic. Cluster similar types of administer if given by an interviewer. questions either by subject, type of response,3. Actual Questions—See Sections on Question and/or instruction. Content and Question Formats. 5. Numbering Sequence—Precoding every item4. Classification Data—Normally these are on the questionnaire allows for ease of data demographic information and respondent entry. However, coding must be done characteristics to ensure the target discretely if it is to appear on all but the population has been sampled adequately. master copy of a written survey to avoid5. Identification Data—This may include names, confusing respondents. addresses, and telephone numbers and/or 6. Transition Statements—When shifting topics identification numbers of participants to keep and/or sections in the questionnaire, clear track of respondents and to facilitate follow- and understandable transition elements or up procedures. statements are important.
Page 5Question ContentThe following factors must be considered when confuseconstructing the questions to be used in the respondents whosurvey instrument: may not correctly interpret the1. Will the question elicit the type of response intended meaning. desired? For example, “How long have you lived in your current home?” An open-ended 7. Is the person question of this type may elicit answers such answering the as “all my life,” instead of number of years. question the appropriate2. Use words which are simple, familiar, and person? Often unambiguous to the target population. Do not times one member of a household has more use colloquialisms or slang. A fifth grade knowledge than others about the reading level should be used when household’s finances. constructing questions. The question “Which detrimental attributes impact on our 8. Will the respondent be willing to answer transportation system?” contains words that certain types of questions truthfully? Some are too difficult. The question “What do kids topics regularly elicit biased responses or in your neighborhood do for fun?” is vague higher item nonresponse. These topics and contains slang. Kids does not define a include information relating to income, or specific age group and can refer to young criminal and other kinds of personal goats. behaviors.3. Avoid double-barrel questions. The question 9. Does the question bias the respondent’s “Do you support smoking policies in private answer? “The president believes Social industry, but not in governmental offices?” is Security should be privatized to protect the two separate questions. funds. Do you agree?” If this appeared on a survey, the answers might reflect feelings4. Determine whether respondents will be able about the president rather than what should to answer accurately; are they likely to know, be done with Social Security. understand, and/or remember items relating to the desired information? Respondent recall 10. Questions which appear to be “off the wall” becomes unreliable quickly. Structure and unrelated to the subject matter being questions to assist memory by measuring a explored should be avoided. The questions discrete and recent time period rather than a should provide the information needed as vague reference to the past. defined in the survey objectives and purpose.6. Avoid questions containing double negatives or phrases such as, “would you agree or 11. The first question is important and should be disagree that…” These types of questions short, simple, and relevant. More complex
Page 6 Survey Questionnaire Design issues can come later in the questionnaire. Questions Should Contain Simple Familiar Words This will ensure higher response rates. Complex Term Simplified Term12. In multiple-choice or close-ended proximity closeness questions, make sure all possible response exhausted tired choices are included and are mutually leisure time free time exclusive. When asking the number of candid honest priority most important times something has occurred, it is not employment work unusual for the answer choice “none” or assistance help “0” to be missing. rectify correctQuestion FormatsThere are five basic formats from which to 4. Closed-ended with unordered responsestructure questions in a survey instrument: choices: “Which of the following job tasks do you like the most?” (circle one letter)1. Open-ended: “The job tasks I enjoy most are___________________________.” A. Writing2. Modified open–ended: “I was _______years B. Editing old when I began my current job.” C. Organizing3. Closed-ended with ordered response 5. Partially close-ended: “Which job task do choices: “How would you rate your prefer- you most enjoy doing?” (circle one letter) ences for the following job tasks?” (circle one answer for each item): A. Writing ANSWER CHOICES B. EditingWriting Enjoy Neutral Dislike C. OrganizingEditing Enjoy Neutral Dislike D. Other (please specify)_____________Organizing Enjoy Neutral Dislike In general, close-ended with ordered or unor- dered response choices are the easiest to code for data processing. Open-ended are the most difficult. However, all question types can be useful depending upon what is being measured (behaviors, attitudes, et cetera) and the kinds of information needed.
Page 7PretestThe last steps before actual to be surveyed. Some administered to ensure:distribution of the preliminary data analysis (even 1. Ease of administration ofquestionnaire include: if hand calculated) should be the survey, attempted to check both1. A review by colleagues and design and coding procedures. 2. Field processes to be potential data users, and It often is useful to run two or employed work smoothly,2. A pretest of the survey more versions of the 3. Questions are easily instrument to be used. questionnaire to determine understood,For comprehensive pretesting, which version will do the best 4. All important questionsa mock copy should be job. This may include have been asked, andsubmitted to a representative variations on questions.cross-sample of the population 5. Instructions are In general, a pretest is understood.Glossary• Bias (error): Distortion or resulting from the survey unreliability in survey results. process. All surveys contain some bias. • Element: The basic unit about Bias is increased when the which survey information is respondents (persons sought (i.e., person, business, answering the survey) are not household, car, dog, et cetera) representative of the population • Instrument: The tool or device being questioned, when used for survey questions are poorly written or measurement, usually misunderstood, and when the a questionnaire. researcher uses inappropriate • Nonresponse: Unit techniques to analyze the data. nonresponse refers to• Census: A study using all the refusal of persons available elements (members) selected to be of a population. sampled to participate• Coding: The assignment of in a survey (i.e., numerical (or other) values to person does not individual questions and return the mail answers on a survey instrument questionnaire). Item (questionnaire) to facilitate nonresponse refers to selected statistical analysis of the questions left unanswered by information. the person responding.• Data: The collection of • Population: The universe or observations and information collection of all elements
Page 8 Common Pitfalls in Conducting a Survey (persons, business, et cetera) being population selected to be sampled. described or measured by a sample. • Sample: Any portion of the population, less• Pretest: An initial evaluation of the survey than the total. design by using a small subsample of the • Survey: A process of inquiry for the purpose intended population for preliminary of data collection and analysis using information. Number of Survey Returned observation, polls, questionnaires, and/or• Questionnaire: A measuring device used to Number of Elements in Sample interviews. query a population/sample in order to • Statistics: Descriptive measures based upon obtain information for analysis. a probability sample. Number of Survey Returned• Response Items: The various answer choices Number of Elements in Sample provided on a survey instrument.• Response Rates: The percentage of surveys/ questionnaires completed from the total sample queried. Typically response rates are calculated as: Surveys Completed Response Rate = Number Sampled - Ineligible Elements × 100 • Respondent: An element or member of the Fairfax County Department of Systems Management for Human Services Economic and Demographic Research 12011 Government Center Parkway Suite 222 Fairfax, Virginia 22035 Phone: 703-324-4519 TTY: 711 (Virginia Relay) Fax: 703-803-8598 Building Upon the Strengths of a Diverse Community We’re on the Web! www.fairfaxcounty.gov/aboutfairfax