The document discusses survey research and questionnaire design, outlining best practices such as avoiding common errors like coverage, sampling, measurement and non-response; asking clear, concise questions; pre-testing questions; and coding and analyzing survey data to produce accurate and timely results that meet the needs of clients. It provides guidance on writing effective questions, ordering questions appropriately, and pre-testing questionnaires to ensure high-quality survey data collection and analysis.

1. Survey Research
Question and Questionnaire Design
David Filiberto, Ph.D.
2013

3. Definition of the Sample Survey
1. Produces quantitative descriptions on aspects of
the study population
2. Asks people questions, written or verbally
3. Information is collected from a sample

4. Advantages (and Disadvantages) of Surveys
Good at…
• Efficient
• Flexible
• Relatively easy to administer
• Economy in data collection
Disadvantages…
• Reliability
• Self – selection bias
• Question design

5. Determinants of a Successful Survey
1. Avoids the four common sources of error:
a) Coverage
b) Sampling
c) Measurement
d) Non –response
2. Produces accurate, timely and accessible data or
“fitness for use” (especially important in private
sector)

6. Beginning of “scientific” or Sample Surveys
Charles Booth: Inquiry into the Set out to answer how many
Life and Labour of the People
in London, 1886-1903 people in London lived in
poverty
• Systematic effort to
interview relevant groups
• Organized research team to
conduct the interviews
• Gathered information over
time and quantified it

7. Beginning of “scientific” or Sample Surveys
Anders Nicolai Kiaer • use of statistical sampling to
(1838-1919) acquire unbiased information
• used the ideas of
stratification, sampling with
unequal probabilities and
multi-stage sampling well
before the development of
any of the statistical theory

8. Best Practices for Survey Research
1. Have specific goals and objectives for the survey
2. Understand and avoid the four types of error
3. Choose the survey method that works best for the project
4. Select samples that represent the population to be studied
5. Write good questions, Order questions effectively
6. Pre test and revise questionnaire
7. Code, computerize, and analyze the data
8. Put together the people and equipment to enact survey in necessary
time frame
9. Present results in a way that is informative to the target audience
10. Maintain perspective while putting plans into action

9. Steps to Developing a Successful Questionnaire
• Decide what information is needed
• Search for existing questions
• Draft new questions
• Order questions effectively
• Pre-test and pilot the questions
• Revise and pre-test until deemed acceptable

10. ‘Five Ws and one H’
(Who, What, Why, When, Where and How)
• What is the specific purpose of the survey?
• What information is the client interested in
exploring?
• Who needs this information, and what are they
going to do with it?
Hint: SOP or client work plan will provide many of the
answers to the questions above

11. Question Wording Basics: Top 10
1. Write short and simple questions
2. Avoid leading questions, wording that influences
respondents to consider a subject in a weighted
manner, or injects a particular preference or opinion
3. Ask close-ended questions. Whenever possible, provide
answer choices and give as few answer choices as is
reasonable. Such questions are much easier to answer
and to analyze after the survey
4. Questions must be non-threatening and attempt to evoke
the truth
5. Use simple language, while not "talking down”

12. Question Wording Basics: Top 10
6. Avoid ambiguities and vague words (e.g.
usual, regular, normal)
7. Don't use double-barreled questions
8. Clearly define the response scale dimension or
continuum
9. Minimize presuppositions - an assumption about the
world whose truth is taken for granted
10. Ask questions on firsthand knowledge. Avoid
hypothetical scenarios

13. Importance of Question Wording
In a national sample, respondents were randomly assigned to be
asked one of two questions:
1. “Do you think the United States should allow public speeches
against democracy?”
2. “Do you think the United States should forbid public speeches
against democracy?”
Results:
Support for free speech is > by more than 20% when respondents
answer question 2 rather than question 1

14. Question Types
5 primary types of questions:
• Behavior - What people do... their actions (What programs do you
watch on TV?)
• Beliefs - What people think... what is true and false for them
(Estimate the percentage of college students who vote
Democratic.)
• Knowledge - The accuracy of people's beliefs (How many college
students vote Democratic in your dorm?)
• Attitudes - what people think is desirable as opposed to true or false.
(Do you think Democrats are doing a good job?)
• Attributes - Characteristics (age, sex, etc.)

15. Principles: Short and Simple
1. Write short and simple questions.
E.g.: “Q. Given the current trend of more hits, more home
runs, longer games in general, and more injuries in
baseball today, do you think that steroid use should
continue to be banned even though it is not enforced?"
*Problem: Long questions can be confusing
Better question:
“Q. Steroid use has both positive and negative effects on
baseball. Do you think that steroid use should be banned?"

16. Principles: No Leading
2. Avoid leading questions - questions which subtly prompt
the respondent to answer in a particular way
E.g.: “Q1. Do you hate the president of the United States?”
Why is this leading? Because the question itself includes an
opinion word.
“Q2. Who do you think of when you hear 9/11?
a. Osama bin Laden
b. Saddam Hussein
c. George W. Bush
Why is this leading?
Because it forces the respondent to answer one of these people, even if
none of them come to mind

17. Principles: Close it Up
3. Ask close-ended questions - closed questions
should have the minimum number of a complete
set of options for the respondent to choose
between. With closed questions, the responses
must be both exhaustive and mutually exclusive
E.g.: Why do you play sports?
1. Enjoyment
2. Health
3. Friends
4. Other-----

18. Principles: Non-threatening
4. Questions must be non-threatening and attempt to
evoke the truth. When a respondent is concerned about
the consequences of answering a question in a particular
manner, there is a good possibility that the answer will
not be truthful
E.g.: “Q. In general, would you say you drink more
than your friends, less than your friends, or about
the same amount as your friends?”

19. Principles: Avoid Condescension
5. Use simple language, while not "talking down”.
Be simple without being condescending:
E.g.: “Q. Should the Surgeon General (i.e., the
head person in charge of health promotion) ban
cigarette smoking?”

20. Principles: Avoid Vagueness and Ambiguity
6. Avoid vague or ambiguous wording - questions should mean the
same thing to all respondents. All the terms should be
understandable or defined, time periods specified, complex
questions asked in multiple stages
E.g., “Q1. How often do you feel tired during the day?”
*To what day are you referring? What time? The answer will drastically
change depending on whether respondent is thinking of a
workday, vacation day, or doing some weird calculus to try to mash
all these days together.
E.g., “Q2. Do you watch television regularly?“
*Vague questions are difficult to answer (what is the meaning of
"regularly"?)
Better question: “Q2. How often do you watch Television?"

21. Principles: Double-barreled
7. Double barreled - Ask one question at a time. Avoid asking 2
questions, imposing unwarranted assumptions, or hidden
contingencies
E.g.: “Q1. With the way the war is going, do you think it’s a good idea to
send more troops?
*This question is double-barreled or imposes an assumption: It asks
people to accept its premise that the war is going badly before they
respond regarding the troops.
Or
“Q2. Are the Cubs and Braves good baseball teams?"
*Problem: Double-barreled questions ask two separate things (perhaps
the Cubs are bad and the Braves are good)
Better question: “Q2. Are the Cubs a good baseball team?"

22. Principles: Define Scales
8. When using a response scale, clearly define the
dimension or continuum respondents are to use in their
rating task
E.g.: Response categories - Make them logical and meaningful:
NOT: Many......Some.......A Few......Very Few.....None
DO a Bipolor or Unipolar rating scale:
Bipolar measures both direction and intensity of an attitude:
Unipolar scale measures one concept with varying degrees of intensity:

23. Principles: Minimize Presuppositions
9. Minimize presuppositions - answering a question implies accepting
its presuppositions, a respondent may be led to provide an answer
even if its presuppositions are false
E.g.: “Q1. Are you a Republican or a Democrat?”
Problem: presupposes that one of the alternatives is true.
Or “Q2. What are your usual hours of work?”
Problem: Does respondent have usual hours of work?
Might be reworded to ask:
“Q2. What are your usual hours of work, or do you not have usual
hours?”

24. Principles: Firsthand Experience
10. Questions should ask for firsthand experiences (if
accuracy is the objective): beware of asking for
secondhand knowledge, hypothetical questions, or
asking about solutions to complex problems
E.g.: “Q. How likely are you to use anesthesia when delivering your baby?”
hypothetical: women who have delivered a baby are better at estimating
their probability of using anesthesia than women who haven’t
delivered a baby.

25. What type of question should I write?

26. Possible Biases when writing questions:
Bias can be a prejudice, flaw, or an attempt to
influence in a question
4 typical biases:
1. Social desirability
2. Memory
3. Positive or negative slant
4. Ordering or context effect

27. Question Order: Serial and Semantic Components
Cognitive processes
– comprehending the question
– retrieving the memories related to it
– judging what information to integrate into the
answer
– finally responding to a response category
Serial – Location in a series of Items
Semantic – Location in a series of Meanings

28. Ordering of Questions
General
1. Opening question non-threatening, broadly worded - early questions
do not restrict what respondents feel they can say later, should be
relevant and easy
2. Questions should be ordered so as to seem logical to the respondent
3. More concrete questions should be placed first, moving to more
abstract questions later
4. Similar questions should be placed in sections to help structure the
survey – thematic clustering
5. Potentially objectionable questions are placed near the end
6. Demographic questions should be placed at the end Specific

29. Organizing the Questionnaire
• Include a cover letter for written surveys, or
verbally enunciate clear purpose and
instructions for telephone and face to face
• Written physical format:
– Use large clear type, take advantage of ‘white space’
– Be consistent with direction and placement of
response categories

30. Pre testing – Why?
Question development
• Make sure that the questions can be clearly understood and have an
adequate range of responses.
• Can eliminate possible errors made by people incorrectly
interpreting the meaning of questions as well as ensuring that there
is enough variation to actually analyze the data.
Questionnaire development
• This allows you to see how people respond to the questionnaire as a
whole.
• Important things to look for are:
– Time
– Flow
– Interest and attention

31. Pre testing – 2.0
• Survey a small, but representative sample of
respondents. Preferable to test the
questionnaires with people like those in your
main study population
• Test questions for biasedness by asking
respondents to guess what the researchers are
predicting for the answers

32. Coding and Analysis
q1, Which best describes your home?
q1_text
• Code the questionnaire 1
2
SINGLE FAMILY, DETACHED HOME
DUPLEX OR TRIPLEX
for quantitative 3
4
ROW HOME
APARTMENT BUILDING
5 MOBILE HOME
analyses - for each 6 TOWN HOME CONDOMINIUM
7 OTHER, please specify_____
question, all possible q2 Approximately when was your home built?
1 BEFORE 1900
responses are assigned 2
3
4
1900 TO 1945
1946 TO 1970
1971 TO 1995
a numerical value 5
6
AFTER 1995
DON’T KNOW
q3 In what type of community is your home located?
1 RURAL
2 VILLAGE
3 SUBURB
4 CITY

33. Coding and Analysis
• Create a dataset (Excel,
SPSS, etc.)
Key in the data for
analysis
• QC

34. Coding and Analysis
Column Valid
• Compute statistics;
Count Column N % N%
1 Which best SINGLE FAMILY, DETACHED
describes your HOME 50 84.7% 86.2%
home?
DUPLEX OR TRIPLEX 8 13.6% 13.8%
ROW HOME 0 .0% .0%
APARTMENT BUILDING 0 .0% .0%
- frequencies, MOBILE HOME
TOWN HOME OR
CONDOMINIUM
0
0
.0%
.0%
.0%
.0%
- measures of central OTHER, please specify________
Missing
0
1
.0%
1.7%
.0%
.0%
tendency Total
86.21
59 100.0% 100.0%
(mean, median mode),
80
- crosstabs,
60
P e rc en t
40
- models to test your
20
13.79
hypothesis or research 0
E
E
E
M
M
M
O
O
O
X
M
G
questions (as
LE
H
H
H
IN
IU
E
W
D
D
IN
IP
IL
E
IL
O
TR
M
H
B
U
R
O
C
O
B
R
D
TA
M
T
O
N
N
E
O
X
E
,D
C
LE
TM
R
Y
appropriate) P
O
IL
R
U
A
M
E
D
P
FA
M
A
O
LE
H
N
G
W
IN
TO
S
1 Which best describes your home?

35. Determinants of Success
1. Avoids the four common sources of error:
Coverage, Sampling, Measurement, Non -response
e.g. Telephone survey response rate 35 to 60%
Mail 35 to 75%
Personal Interview 60 to 80%
2. Produces accurate, timely and accessible data or
“fitness for use” for constituent agencies

36. Thank You
David Filiberto
dmf22@cornell.edu