This document provides guidance on effective survey design. It discusses what surveys can measure, including behaviors, attitudes, demographics, and preferences. It also describes different types of survey questions such as open-ended, closed-ended, rating scales, rankings, and checklists. Common problems with question wording like bias, unequal comparisons, and threats to self-esteem are explained. The document recommends using question scales that directly measure the concept in question rather than artificial agreement scales. It emphasizes the importance of pilot testing surveys and refining questions based on feedback.

1. The Do’s and Don’ts
of Survey Design
Tips, tricks, and techniques for
making your survey research
more effective

2. What do surveys measure?
 Surveys






measure characteristics of people
Behaviors
Attitudes
Beliefs
Opinions
Demographics
Sociographics

3. What do surveys measure?
 Surveys

measure characteristics of people
Behaviors
Actions that are directly observable by others.

4. What do surveys measure?
 Surveys

measure attributes of people:
Behaviors
Actions that are directly observable by others.

Demographics
Physical characteristics that are directly observable
and directly verifiable.

5. What do surveys measure?
 Surveys

measure attributes of people:
Behaviors
Actions that are directly observable by others.

Demographics
Physical characteristics that are directly observable
and directly verifiable.

Sociographics
Social characteristics that are directly observable but
not directly verifiable (i.e., requires confirmation).

6. What do surveys measure?
 Surveys

Attitudes, Opinions, Beliefs
 Surveys

measure preferences of people:
Wants and needs
 Surveys

measure mindsets of people:
measure constructs of people:
Satisfaction, Confidence, Persistence, etc.

7. Types of survey questions







Open response
Partially open response
Closed response
Semantic differential scales
Agreement and rating scales
Ranking scales
Checklists

8. Open response
What was your entry status when you
started your first semester at UNF?
____________________________

9. Partially open
response
What was your entry status when you
started your first semester at UNF?
1.
2.
3.
RECENT HIGH SCHOOL GRADUATE
TRANSFERRED FROM COMMUNITY
COLLEGE OR OTHER INSTITUTION
OTHER________________________

10. Closed
If you could do it all over again,
would you choose to come to
UNF?
1.
2.
3.
YES, I WOULD CHOOSE UNF
NO, I WOULD CHOOSE ANOTHER
UNIVERSITY
NO, I WOULD NOT GO TO
COLLEGE

11. Semantic differential
On a scale of 1 to 5, please indicate
the quality level of your instructors
by checking one of the boxes below:
11
 The Best



11
 The Worst
11
11
11

12. Agreement scale
The quality of my instructors is
excellent.
1.
2.
3.
4.
5.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree

13. Rating scale
Overall, how would you rate the
quality of your instructors?
1.
2.
3.
4.
5.
Very Good
Good
Fair
Poor
Very Poor

14. Ranking scale
Write the letter of each instructor in
the numbered spaces below in order
of their relative quality:
A.
B.
C.
D.
E.
Instructor Jones
Instructor Smith
Instructor Brown
Instructor Lane
Instructor White
1. __________
2. __________
3. __________
4. __________
5. __________

15. Checklist
In which activities did you
participate? (Check all that apply):
 STUDENT GOVERNMENT
 POLITICAL ACTIVITIES
 INTRAMURAL SPORTS
 INTERCOLLEGIATE ATHLETICS
 COMMUNITY SERVICE

16. Why people answer surveys
 Theory
of Social Exchange
 Perceived importance
 Costs and rewards
 Established trust
 Anonymity
 Peer pressure

17. Common wording problems
Bias








Objectionable questions
Unequal comparisons
Bias in more than one direction
Unbalanced categories
Emotionally charged words
Threats to self-esteem
Personalization

18. Common wording problems

Objectionable questions


How old are you? ___________ (WRONG)
Please indicate your age range: (RIGHT)
a. 21 and under
b. 22-35
c. 36-49
d. 50 and over

19. Common wording problems

Unequal comparisons

Which of the following poses the greater threat
to peace in the Middle East? (WRONG)
a. Palestinian attacks against Israeli citizens
b. Israeli attacks against Palestinian militants
c. Both a and b threaten peace

Which of the following must cease for there to
be peace in the Middle East? (RIGHT)
a. Palestinian attacks against Israeli citizens
b. Israeli attacks against Palestinian militants
c. Both a and b must cease

20. Common wording problems

Bias in more than one direction

Are you against same sex marriage and in favor of a
constitutional amendment to ban it? (WRONG)
a. Yes
b. No
c. Unsure

What is your view on same sex marriage? (RIGHT)
a. I think marriage is a matter of personal choice
b. I’m against it but don’t want a constitutional amendment
c. I want a constitutional amendment banning it

21. Common wording problems

Unbalanced categories

Please indicate your age range: (WRONG)
a. 18-20
b. 21-30
c. 31-50
d. 51 and over

Please indicate your age range: (RIGHT)
a. 21 and under
b. 22-35
c. 36-49
d. 50 and over

22. Common wording problems

Emotionally charged words

Was the FDC negligent by ignoring the warnings about
Vioxx during testing and approving it for sale? (WRONG)
a. Yes
b. No
c. Unsure

If the FDC knew that Vioxx caused serious side effects
during testing, what should it have done? (RIGHT)
a. Ban it from ever being sold.
b. Require more testing before approving it
c. Unsure

23. Common wording problems

Threats to self-esteem

How often have you driven drunk? (WRONG)
a. Never
b. Once
c. Two or more times

Have you ever driven a vehicle after you have
had too much to drink? (RIGHT)
a. Never
b. Yes, on one occasion
c. Yes, on more than one occasion

24. Common wording problems

Personalization

Dear Mr. Jones: How would you rate the level of service
that you received from Salesman Mr. Smith? (WRONG)
a. Excellent
b. Good
c. Fair
d. Poor

Dear Valued Customer: How would you rate the level of
service that you received from your salesperson? (RIGHT)
a. Excellent
b. Good
c. Fair
d. Poor

25. The problem with scales of agreement
Artificial position statements are made to fit these formats.
“Agree/Disagree” statements require respondents to
take a position “For” or “Against” an issue.
There is no clear-cut way to interpret one’s agreement
or disagreement with a statement when the proper
response choices are unrelated to agreement.
Too many items using agreement scales are built upon
“face validity” alone. The underlying assumption is that
agreeing with an item indicates a person has a specific
opinion about that item.
“Agreement-disagreement” is not a semantic
differential.

26. Agree-Disagree is not a
semantic differential scale!
 Agreeing
with a statement worded positively
may have a different meaning than
disagreeing with the same statement worded
negatively.
True semantic differential items:
Hot
Weak
Very Good
Frequently
Very Helpful
Cold
Strong
Very Poor
Never
Not Helpful at all

27. 1. There should be designated
parking areas for students.
SA
A
N
D
SD
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
“I shouldn’t have to
waste time looking
for parking spaces!”
What was the student thinking when he chose “Agree?”

28. 1. There should be designated
parking areas for students.
SA
A
N
D
SD
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
“I should have a
place to park just
like faculty have!”
not
What was the student thinking when he chose “Disagree?”

29. You can improve survey items by
using a more direct statement of
outcome and a more direct scale
of measurement.

30. Here is a common example
taken from the ISQ survey:

Professor clearly explains complex concepts and ideas.
Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
No response
And now….a better way to ask this question:
How would you describe your professor’s explanations of
complex concepts and ideas?

1.
2.
3.
4.
5.
Always clear and concise
Mostly clear and concise
Sometimes clear and concise & other times obscure and vague
Mostly obscurely and vague
Always obscure and vague

31. The problem with scales of
satisfaction:
To classify data as "satisfaction information," survey items
are often “shoehorned” into a Satisfaction-Dissatisfaction
format. These items use the same scale format for expediency
sake only, and not because the scale fits the meaning of the
statement.
Satisfaction surveys are good at identifying general problem
areas, but do not lend themselves to specific solutions.
All of the statements are typically couched in positive terms
thus increasing the likelihood of a positive response bias.
In a vague attempt to not be overly positive, many items
overuse the words “Adequate” and “Reasonable.”

32. Keys to making your surveys
better and more useful
Use a scale covering the whole range of responses.
Don’t be afraid of criticism. Make your items balanced.
Use a scale or response list that matches the statement
Identify a potential problem first, and then use survey
data to define its scope and suggest possible
solutions.
Use a survey only for its intended purpose.
Refine your surveys each time you use them.
Don’t forget to include free response input.

33. The Do’s and Don’ts
of Survey Design
The End