Survey question and questionnaire design slideshare 022113 dmf
Survey ResearchQuestion and Questionnaire Design David Filiberto, Ph.D. 2013
Definition of the Sample Survey 1. Produces quantitative descriptions on aspects of the study population 2. Asks people questions, written or verbally 3. Information is collected from a sample
Advantages (and Disadvantages) of Surveys Good at… • Efficient • Flexible • Relatively easy to administer • Economy in data collection Disadvantages… • Reliability • Self – selection bias • Question design
Determinants of a Successful Survey1. Avoids the four common sources of error: a) Coverage b) Sampling c) Measurement d) Non –response2. Produces accurate, timely and accessible data or “fitness for use” (especially important in private sector)
Beginning of “scientific” or Sample SurveysCharles Booth: Inquiry into the Set out to answer how many Life and Labour of the People in London, 1886-1903 people in London lived in poverty • Systematic effort to interview relevant groups • Organized research team to conduct the interviews • Gathered information over time and quantified it
Beginning of “scientific” or Sample Surveys Anders Nicolai Kiaer • use of statistical sampling to(1838-1919) acquire unbiased information • used the ideas of stratification, sampling with unequal probabilities and multi-stage sampling well before the development of any of the statistical theory
Best Practices for Survey Research 1. Have specific goals and objectives for the survey 2. Understand and avoid the four types of error 3. Choose the survey method that works best for the project 4. Select samples that represent the population to be studied 5. Write good questions, Order questions effectively 6. Pre test and revise questionnaire 7. Code, computerize, and analyze the data 8. Put together the people and equipment to enact survey in necessary time frame 9. Present results in a way that is informative to the target audience 10. Maintain perspective while putting plans into action
Steps to Developing a Successful Questionnaire• Decide what information is needed• Search for existing questions• Draft new questions• Order questions effectively• Pre-test and pilot the questions• Revise and pre-test until deemed acceptable
‘Five Ws and one H’ (Who, What, Why, When, Where and How)• What is the specific purpose of the survey?• What information is the client interested in exploring?• Who needs this information, and what are they going to do with it?Hint: SOP or client work plan will provide many of the answers to the questions above
Question Wording Basics: Top 10 1. Write short and simple questions 2. Avoid leading questions, wording that influences respondents to consider a subject in a weighted manner, or injects a particular preference or opinion 3. Ask close-ended questions. Whenever possible, provide answer choices and give as few answer choices as is reasonable. Such questions are much easier to answer and to analyze after the survey 4. Questions must be non-threatening and attempt to evoke the truth 5. Use simple language, while not "talking down”
Question Wording Basics: Top 10 6. Avoid ambiguities and vague words (e.g. usual, regular, normal) 7. Dont use double-barreled questions 8. Clearly define the response scale dimension or continuum 9. Minimize presuppositions - an assumption about the world whose truth is taken for granted 10. Ask questions on firsthand knowledge. Avoid hypothetical scenarios
Importance of Question WordingIn a national sample, respondents were randomly assigned to be asked one of two questions: 1. “Do you think the United States should allow public speeches against democracy?” 2. “Do you think the United States should forbid public speeches against democracy?”Results: Support for free speech is > by more than 20% when respondents answer question 2 rather than question 1
Question Types 5 primary types of questions: • Behavior - What people do... their actions (What programs do you watch on TV?) • Beliefs - What people think... what is true and false for them (Estimate the percentage of college students who vote Democratic.) • Knowledge - The accuracy of peoples beliefs (How many college students vote Democratic in your dorm?) • Attitudes - what people think is desirable as opposed to true or false. (Do you think Democrats are doing a good job?) • Attributes - Characteristics (age, sex, etc.)
Principles: Short and Simple1. Write short and simple questions. E.g.: “Q. Given the current trend of more hits, more home runs, longer games in general, and more injuries in baseball today, do you think that steroid use should continue to be banned even though it is not enforced?" *Problem: Long questions can be confusing Better question: “Q. Steroid use has both positive and negative effects on baseball. Do you think that steroid use should be banned?"
Principles: No Leading2. Avoid leading questions - questions which subtly prompt the respondent to answer in a particular way E.g.: “Q1. Do you hate the president of the United States?” Why is this leading? Because the question itself includes an opinion word. “Q2. Who do you think of when you hear 9/11? a. Osama bin Laden b. Saddam Hussein c. George W. Bush Why is this leading? Because it forces the respondent to answer one of these people, even if none of them come to mind
Principles: Close it Up3. Ask close-ended questions - closed questions should have the minimum number of a complete set of options for the respondent to choose between. With closed questions, the responses must be both exhaustive and mutually exclusive E.g.: Why do you play sports? 1. Enjoyment 2. Health 3. Friends 4. Other-----
Principles: Non-threatening4. Questions must be non-threatening and attempt to evoke the truth. When a respondent is concerned about the consequences of answering a question in a particular manner, there is a good possibility that the answer will not be truthful E.g.: “Q. In general, would you say you drink more than your friends, less than your friends, or about the same amount as your friends?”
Principles: Avoid Condescension5. Use simple language, while not "talking down”. Be simple without being condescending: E.g.: “Q. Should the Surgeon General (i.e., the head person in charge of health promotion) ban cigarette smoking?”
Principles: Avoid Vagueness and Ambiguity6. Avoid vague or ambiguous wording - questions should mean the same thing to all respondents. All the terms should be understandable or defined, time periods specified, complex questions asked in multiple stages E.g., “Q1. How often do you feel tired during the day?” *To what day are you referring? What time? The answer will drastically change depending on whether respondent is thinking of a workday, vacation day, or doing some weird calculus to try to mash all these days together. E.g., “Q2. Do you watch television regularly?“ *Vague questions are difficult to answer (what is the meaning of "regularly"?) Better question: “Q2. How often do you watch Television?"
Principles: Double-barreled7. Double barreled - Ask one question at a time. Avoid asking 2 questions, imposing unwarranted assumptions, or hidden contingencies E.g.: “Q1. With the way the war is going, do you think it’s a good idea to send more troops? *This question is double-barreled or imposes an assumption: It asks people to accept its premise that the war is going badly before they respond regarding the troops. Or “Q2. Are the Cubs and Braves good baseball teams?" *Problem: Double-barreled questions ask two separate things (perhaps the Cubs are bad and the Braves are good) Better question: “Q2. Are the Cubs a good baseball team?"
Principles: Define Scales8. When using a response scale, clearly define the dimension or continuum respondents are to use in their rating task E.g.: Response categories - Make them logical and meaningful: NOT: Many......Some.......A Few......Very Few.....None DO a Bipolor or Unipolar rating scale: Bipolar measures both direction and intensity of an attitude: Unipolar scale measures one concept with varying degrees of intensity:
Principles: Minimize Presuppositions9. Minimize presuppositions - answering a question implies accepting its presuppositions, a respondent may be led to provide an answer even if its presuppositions are false E.g.: “Q1. Are you a Republican or a Democrat?” Problem: presupposes that one of the alternatives is true. Or “Q2. What are your usual hours of work?” Problem: Does respondent have usual hours of work? Might be reworded to ask: “Q2. What are your usual hours of work, or do you not have usual hours?”
Principles: Firsthand Experience10. Questions should ask for firsthand experiences (if accuracy is the objective): beware of asking for secondhand knowledge, hypothetical questions, or asking about solutions to complex problems E.g.: “Q. How likely are you to use anesthesia when delivering your baby?” hypothetical: women who have delivered a baby are better at estimating their probability of using anesthesia than women who haven’t delivered a baby.
Possible Biases when writing questions: Bias can be a prejudice, flaw, or an attempt to influence in a question 4 typical biases: 1. Social desirability 2. Memory 3. Positive or negative slant 4. Ordering or context effect
Question Order: Serial and Semantic Components Cognitive processes – comprehending the question – retrieving the memories related to it – judging what information to integrate into the answer – finally responding to a response category Serial – Location in a series of Items Semantic – Location in a series of Meanings
Ordering of Questions General 1. Opening question non-threatening, broadly worded - early questions do not restrict what respondents feel they can say later, should be relevant and easy 2. Questions should be ordered so as to seem logical to the respondent 3. More concrete questions should be placed first, moving to more abstract questions later 4. Similar questions should be placed in sections to help structure the survey – thematic clustering 5. Potentially objectionable questions are placed near the end 6. Demographic questions should be placed at the end Specific
Organizing the Questionnaire • Include a cover letter for written surveys, or verbally enunciate clear purpose and instructions for telephone and face to face • Written physical format: – Use large clear type, take advantage of ‘white space’ – Be consistent with direction and placement of response categories
Pre testing – Why? Question development • Make sure that the questions can be clearly understood and have an adequate range of responses. • Can eliminate possible errors made by people incorrectly interpreting the meaning of questions as well as ensuring that there is enough variation to actually analyze the data. Questionnaire development • This allows you to see how people respond to the questionnaire as a whole. • Important things to look for are: – Time – Flow – Interest and attention
Pre testing – 2.0 • Survey a small, but representative sample of respondents. Preferable to test the questionnaires with people like those in your main study population • Test questions for biasedness by asking respondents to guess what the researchers are predicting for the answers
Coding and Analysis q1, Which best describes your home? q1_text• Code the questionnaire 1 2 SINGLE FAMILY, DETACHED HOME DUPLEX OR TRIPLEX for quantitative 3 4 ROW HOME APARTMENT BUILDING 5 MOBILE HOME analyses - for each 6 TOWN HOME CONDOMINIUM 7 OTHER, please specify_____ question, all possible q2 Approximately when was your home built? 1 BEFORE 1900 responses are assigned 2 3 4 1900 TO 1945 1946 TO 1970 1971 TO 1995 a numerical value 5 6 AFTER 1995 DON’T KNOW q3 In what type of community is your home located? 1 RURAL 2 VILLAGE 3 SUBURB 4 CITY
Coding and Analysis• Create a dataset (Excel, SPSS, etc.) Key in the data for analysis• QC
Coding and Analysis Column Valid• Compute statistics; Count Column N % N% 1 Which best SINGLE FAMILY, DETACHED describes your HOME 50 84.7% 86.2% home? DUPLEX OR TRIPLEX 8 13.6% 13.8% ROW HOME 0 .0% .0% APARTMENT BUILDING 0 .0% .0% - frequencies, MOBILE HOME TOWN HOME OR CONDOMINIUM 0 0 .0% .0% .0% .0% - measures of central OTHER, please specify________ Missing 0 1 .0% 1.7% .0% .0% tendency Total 86.21 59 100.0% 100.0% (mean, median mode), 80 - crosstabs, 60 P e rc en t 40 - models to test your 20 13.79 hypothesis or research 0 E E E M M M O O O X M G questions (as LE H H H IN IU E W D D IN IP IL E IL O TR M H B U R O C O B R D TA M T O N N E O X E ,D C LE TM R Y appropriate) P O IL R U A M E D P FA M A O LE H N G W IN TO S 1 Which best describes your home?
Determinants of Success1. Avoids the four common sources of error: Coverage, Sampling, Measurement, Non -response e.g. Telephone survey response rate 35 to 60% Mail 35 to 75% Personal Interview 60 to 80%2. Produces accurate, timely and accessible data or “fitness for use” for constituent agencies