Provide a 200 word or more discussion on a topic related to the EP.docx

Provide a 200 word or more discussion on a topic related to the
EPA Safer Choice Program, eco-labels, and risk or hazard in the
chemical industry. Your discussion could include examples not
covered in the lecture videos. Be sure your discussion topic is
pertinent to the principles of green chemistry.
1
Extra Information on Developing Questions
Question Development is the practice of selecting appropriate
response formats and wording
questions that are understandable, unambiguous, and unbiased.
Question development is
absolutely vital to the success of survey. My other handouts
related to 8 have dealt with how to
select appropriate response formats and this handout focuses on
wording questions.
Look at several examples below to appreciate how wording can
introduce bias.
Are you trying to control your compulsive gambling?
___ Yes ___ No

If you answer “Yes” or “No,” you are admitting to a gambling
addiction. Either way, the
conclusion is that everyone who took part in the survey gambles
compulsively!
Did you see the broken headlight?
Did you see a broken headlight?
The question containing “the” produced more “don’t know” and
“yes” answers than did the
question containing “a.”
Did you ever use a Laundromat?
Did you have any concerns about your cell phone’s reception?
Do you always buy from Wal-Mart?
“Ever” means anytime in your lifetime. “Any” means absolutely
the tiniest concern. “Always”
means every time without fail. These commonly used words are
extreme absolutes, meaning that
they place respondents in a situation where they must either
agree fully or they must completely
disagree with the extreme position in the question.
How many and what brands of aspirin did you see last time
when you bought some?
It is nonsensical to ask respondents about details they don’t

recall.
What is the price per gallon of premium gasoline at the Exxon
station on the corner?
This question invites respondents to guess.
How often would you frequent this new, upscale restaurant that
will be built 10 miles from your
home?
This question asks respondents to predict their actions in
circumstances they cannot fathom.
2
Four guidelines for question wording (four DOs of question
wording):
ld be focused on a single issue or topic.
Four DON’Ts of question wording:
wer.

-barreled” question.
A leading question gives a strong cue or expectation as what
answer to provide.
A leading question is typically obvious, whereas a loaded
question is less obvious. A loaded
question has buried in its wording elements a sneaky
presupposition, or it might make reference
to universal beliefs or rules of behavior. It may even apply
emotionalism or touch on a person’s
inner fears.
Look at the following examples.
What type of hotel do you usually stay in when on a trip?
This question is not focused because it does not narrow down
the type of trip or when the hotel is
being used.
A better version: When you are on a family vacation, what type
of destination hotel do you
typically use?
When do you typically go to work?
This question is not focused. Does it mean when do you leave

home for work or when do you
actually begin work once at your workplace?
A better version: At what time do you ordinarily leave home for
work?
What are the considerations that would come to your mind while
you are confronted with the
decision to have some type of repair done on the automatic
icemaker in your refrigerator
assuming that you noticed it was not making ice cubes as well
as it did when you first bought
it?
This question is not brief.
A better version: If your icemaker was not working right, how
would you correct the problem?
3
If you were looking for an automobile that would be used by the
head of your household who
is primarily responsible for driving your children to and from
school, music lessons, and
friends’ houses, how much would you and your spouse discuss
the safety features of one of

the cars you took for a test drive?
This question is not grammatically simple.
A better version: Would you and your spouse discuss the safety
features of a new family car? (if
yes) Would you discuss safety “very little,” “some,” “a good
deal,” or “to a great extent”?
How many children do you have?
This question is unclear because it can be interpreted in several
ways. One respondent may think
of only those children living at home, whereas another might
include children from a previous
marriage.
A better version: How many children under the age of 18 live
with you in your home?
As a Toyota owner, you are satisfied with your car, aren’t you?
This is a leading question. “Aren’t you” needs to be removed.
Have you heard about the satellite radio system that everyone is
talking about?
This is a leading question. “Everyone” is leading. A respondent
may say “yes” to avoid looking
stupid.

Do you think that prices will continue climbing?
This question has built-in assumption that price will rise so is
leading.
To what extent do you agree that Las Vegas is a gambling
paradise?
This question explicitly asks for agreement or disagreement as
opposed to asking if the
respondent agrees or disagrees. This question assumes
respondents all agree but some may not.
Many people think that the price of a Lexus is unaffordable.
Don’t you?
This is a leading question. “Don’t you” is leading.
Don’t you agree with patriotic Americans that buying foreign
automobile models hurts the
U.S. economy?
This question is loaded. “Don’t you” and “patriotic” are loaded.
4
The 2010 BP oil spill caused havoc in the Gulf of Mexico. How
do you feel about BP’s

ability to prevent a similar disaster?
This question is loaded. It poses an emotional or evaluative
condition and then ask the
question.
Should people be allowed to protect themselves from harm by
using a Taser in self-defense?
This question is loaded. Most of the respondents if not all will
answer “yes” to this question
so it is quite biased. A better version: Do you think carrying a
Taser is acceptable for
someone who believes it is needed?
Were you satisfied with the restaurant’s food and service?
Double-barreled question. You need to break it down and make
it two separate questions –
one asking for “food” and one asking for “service.”
How much do you think you would pay for a pair of sunglasses
that will protect your eyes
from the sun’s harmful ultraviolet rays, which are known to
cause blindness?
Overstated question. A better version: How much would you
pay for sunglasses that will
protect your eyes from the sun’s glares?

The following questions are about automobile GPS system.
Be focused. A better version: Please rate your automobile GPS
system on each of the
following features. (Features are listed)
your automobile’s
GPS system to find the fastest way to work?
Be brief. A better version: Does automobile GPS system help
you arrive at work on time?
over 10 miles from your
house for your child to attend a birthday party, would you rely
on your automobile
GPS system to get you there?
Be simple and structured. A better version: To what extent
would you rely on your
automobile GPS system to find a friend’s house?
5

Be clear. A better version: How useful is your automobile GPS
system for each of the
following occasions? (Occasions are listed)
Don’t lead. A better version: In your opinion, how helpful is an
automobile GPS system?
ms were shown to help us decrease our depletion
of world oil reserves,
would you purchase one?
Don’t load. A better version: How much do you an automobile
GPS system might save
you on gasoline?
em if
it saved you time,
money and worry?
Don’t use double-barreled question. A better version: Would
you consider buying an
automobile GPS system if you believed it would reduce your
commuting time by 10%?
an help you avoid
traffic jams that may

last for hours?
Don’t overstate. A better version: To what extent do you believe
an automobile GPS
system will help you avoid traffic congestion?
1
Questionnaire Design
The use of a formal questionnaire helps ensure the
comparability of the data collected across both
interviewers and respondents and, as a result, permits statistical
analysis of this data. Properly
designed questionnaires increase the speed with which the data
can be collected and promote greater
accuracy in the recording and handling of the data. For
example, good questionnaire design can
contribute to the ease of administration, editing, and coding of
the questionnaire. More importantly,
good questionnaire design is critical to the validity of the data
collected. For large scale surveys (that
is, sample sizes of 500 or more), the error due to poorly worded
questions is likely to be much
greater than that of the random sampling error accounted for in

confidence intervals.
Novices at questionnaire design – and too frequently those with
a moderate amount of experience –
tend to falsely assume that if subjects complete the
questionnaire their responses are meaningful.
Unfortunately, respondents will typically select a response to
the most meaningless of multiple
choice questions without any hesitation and never say anything
to the interviewer about the fact that
the question made no sense to them. For example, one study in
the late 1950s asked respondents the
following question:
Do you approve or disapprove of the Metallic Metals Act of
1956?
Approve ______ Disapprove ______ No Opinion ______
Seventy percent of the sample expressed an opinion, either
approval or disapproval of this
legislation even though it never existed.
Questionnaire Design Process
Following is an outline of the questionnaire design process.
Each of these steps will be dealt with in
detail in the following sections:
1) Specify information needed
2) Specify type of interviewing method
3) Determine question content
4) Choose question structure
5) Choose question wording
6) Determine question sequence (order)

7) Design form/layout/code for data entry
8) PRETEST-PRETEST-PRETEST!!!
Specify Information Needed
The first step in questionnaire design is to specify the
information needed. Note that this was also
the first and most important step in the research design process
– as the research project progresses,
the information needed becomes more and more clearly defined.
It is helpful at this stage to review
the components of the managerial and research problem, the
hypotheses, sample, research design,
2
etc., to help focus your thinking as you prepare to create a
questionnaire. The researcher must then
sit down and outline the exact information that s/he wants to
collect from this survey instrument.
Just as important at this stage as deciding what information will
be collected are the decisions about
how that information will be analyzed. The researcher should
create a “dummy table” at this stage –
a dummy table is a blank table used to catalog data as it will be
entered into the analysis software. It
describes how the data will be structured once the data have
been collected and will help the
researcher determine how the information can be analyzed.
Without an analysis plan in place

PRIOR to creating and administering the questionnaire, it is
likely the data will be collected in forms
that prove to be unusable in testing the required hypotheses.
Specify Type of Interviewing Method
The type of interviewing method – mail, telephone, personal or
internet – was addressed in the
reading Chapter 10. Review that chapter for help in deciding on
the type of method to use for a
particular information need.
Determine Question Content
The content of each question to be included in the questionnaire
should be guided by the
specification of information needs in the problem definition
stage, i.e., your investigative
questions. Obviously, if the specification of information needs
is not sufficiently precise, it will be
difficult to determine which questions are necessary and what
their exact content should be.
One of the objectives of good questionnaire design is to write
the questions to overcome
respondents’ unwillingness to answer and their inability to
answer accurately. The following
discussions of question content, structure and wording all have
an impact on respondents’ ability
and willingness to respond accurately.
There are several factors one should consider in designing the
content of the questionnaire:

Is a particular question that has been proposed for inclusion in
the questionnaire really
necessary? Managers frequently want to include questions they
feel are "interesting" but don't know
exactly how they will analyze, or use them once the data are
collected. Somehow they imagine that
once they see the results of these questions they will know what
to do with them. Unfortunately, it
rarely happens that the managerial relevance of including a
particular question becomes clearer after
the data are collected. Instead, the reverse is likely to be the
case because in most studies the results
usually do not strongly favor one alternative course of action
over another.
There tends to be an incorrect perception that a few extra
unnecessary questions will not
increase the cost of the study very much and that one doesn't
need to worry about how all of the
questions are going to be analyzed before the survey is
conducted. Although it is true that adding a
few extra questions to the questionnaire may not increase the
field costs (printing, mailing, and
interviewer costs) very much, there are other "costs" to be
considered. The amount of time and
effort a respondent will devote to a particular survey does not
increase proportionately with the
length of the questionnaire. For example, if respondents would
spend an average of 5 minutes with a
one-page questionnaire, they are likely to spend perhaps only 8
minutes with a two page
questionnaire on the same general topic. This suggests that
respondents will give less thought to

each question and, therefore, provide less accurate data.
3
Also, the increased size of the questionnaire is likely to reduce
the response rate, particularly for
mail surveys, and thereby increase the cost per completed
interview and raise the level of non-
response bias. Finally, including unnecessary questions is likely
to create confusion in analyzing the
results and increase the length of the final report. For example,
with 10 questions there are C(10,2)
= 45 possible two-way cross classification tables one could
construct compared with C(20,2) = 190
for a 20 item questionnaire.
Does the question relate to the precise issue of interest? Not
infrequently, the question as
worded does not relate to the exact issue of interest. For
example, if you were to ask respondents
where they most frequently shopped for clothes, they might
specify fairly high priced specialty
stores. They may shop there to find out what is in style and to
look for sale merchandise. However,
they may buy most of their clothes at stores like JC Penney's
because that store’s clothes are more
affordable.
Is more than one question required? Sometimes it is necessary
to use a "screening

question" to determine if a particular questionnaire item is
meaningful for the respondents to
answer. For example, one should not ask respondents what
brand of computer they own without
first determining that they own one.
Occasionally, researchers will formulate a question that
contains multiple issues. For
example, consider the following item:
Do you think Tide gets clothes clean without injuring the
fabric?
Yes ______ No ______
If consumers answered "No" to this item, there would be no way
of determining whether it
was the "clean" issue, the "injury" issue, or both that prompted
them to do so.
Can the respondent be expected to answer the question
accurately?
Questions that may seem unreasonable to the respondent should
be avoided, or, if /they are
important, the respondent should be provided with an
explanation for their inclusion.
Most respondents don't understand why interviewers inquire
about such things as their education or
income in a survey that was presented as being focusing on

some product or service. Furthermore,
many will refuse to respond to items of this type unless they are
provided with a short but clear
explanation of the interviewer's need to collect this data (e.g.,
“I would like to ask you a few
questions about yourself to help us describe the participants in
our sample”). When these questions
are asked they should come last in the questionnaire and this
data should only be collected at the
level of detail necessary for the study. For example, if in the
data analysis income is to be divided
into three groups (e.g., less than $10,000/year, between $10,000
and $29,999/year, and $30,000/year
or more), these groupings should be used as response categories
in the questionnaire rather than
asking respondents for more specific (and thus more personal)
descriptions of their income.
4
Determine Question Structure
The structure of questions can have a great deal of influence on
whether subjects are able and
willing to answer accurately.
Questions can be either unstructured or structured.
Unstructured questions are those that are open-
ended, in which respondents are expected to fill in a blank by

answering in their own words.
Examples of unstructured questions include:
What is your occupation?
What do you think of people who patronize discount clothing
stores?
Who is your favorite political figure?
Unstructured questions should be very limited in survey
research. By the stage in the research
process in which descriptive research is appropriate, the
researcher should be past the exploratory
phase and should have very well defined information needs that
can be collected in structured
question formats. However, unstructured questions might be
useful as first questions on a topic in a
personal or telephone interview (however, never include them
as a first question on a written survey
– they require too much “work” on the part of respondents and
will greatly increase non-response
rates). As an opening question during an interview,
unstructured questions allow respondents to
express general attitudes and opinions that can help researchers
interpret responses to structured
questions. However, the potential for interviewer or interpreter
bias is high on unstructured
questions, just as it is for exploratory interviewing techniques.
Coding of responses is time-
consuming and costly. Unstructured questions also give extra
weight to the opinions of respondents
who are more articulate or more motivated to respond. Most
importantly, recognize that
unstructured questions are rarely appropriate for written survey
techniques – either mail

questionnaires or email/internet surveys. Even if subjects do
respond to them, people tend to
be much more brief in writing than in speaking. More
importantly, self-administered (written)
surveys have much higher refusal rates for participation given
the additional effort perceived in
answering unstructured questions. If you find yourself
designing a questionnaire that requires
a large number of unstructured questions, you probably ought to
back up and do more
exploratory research before attempting a survey.
In contrast, structured questions provide some sort of organized
response categories. Structured
question response categories can be dichotomous, multiple-
choice or scales. A dichotomous
question is one with two possible answers, for example:
Are you the primary purchaser of groceries in your household?
____ Yes
____ No
Some examples of typical multiple-choice questions include:
What was your annual household income in 2000, from all
sources, before taxes?
____ Less than $20,000

5
____ $20,000 to $39,999
____ $40,000 to $59,999
____ $60,000 to $79,999
____ $80,000 to $99.999
____ $100,000 or more
Which of the following grocery stores have you purchased
products in within the last
month? (Check all that apply)
____ AJ’s Fine Foods
____ Basha’s
____ Fred Meyers
____ Fry’s
____ Safeway
____ Whole Foods Market
____ Other (Please list here: ________________________)
Do you intend to buy a car within the next six months?
____ Definitely will not buy
____ Probably will not buy
____ Undecided
____ Probably will buy
____ Definitely will buy
Scales have been dealt with in detail in the “Measurement and
Scaling” portion of this
course, and are described in the “Scaling Techniques” reading

section. Review those
readings again if necessary. Therefore, following is a more
detailed discussion of issues regarding
the design of dichotomous or multiple-choice structured
questions.
When using multiple choice questions it is important that the
response categories be mutually
exclusive and collectively exhaustive. The mutually exclusive
requirement specifies that there can
be no overlap in meaning between any of the response
categories. The collectively exhaustive criteria
stipulates that all meaningful responses be listed. For example,
consider the following question.
How do you most frequently get to work?
____ personal car
____ motorcycle
____ car pool
____ public transportation
____ walk
____ bus
These responses are not mutually exclusive because a bus is a
form of public transportation.
Some respondents who most frequently use the bus to get to
work will check "public
transportation" while others will check "bus" – the analyst will
have no way of telling how many of
those who checked "public transportation" took the bus and how
many used other forms of public

6
transportation. Furthermore, some respondents may most
frequently ride a bicycle, a response that
is not listed. Therefore, the response categories also are not
collectively exhaustive.
One can use an "other (please specify)" category to overcome
the problem of not having a
collectively exhaustive set of response categories. However,
past research has shown that
respondents are reluctant to check this category. That is, there
is a strong bias against those
alternatives that are not specified in the list. If in the above
example, an "other (please specify)"
category were added, only about half of those respondents who
rode bicycles to work would be
likely to check this category and write in that they rode bicycles
to work. In some sense it is quite
like being a "write-in" candidate in an election.
Another issue relating to dichotomous or multiple choice
questions is the order in which the
responses are listed. There is a strong bias in favor of the first
alternative listed when the responses
are verbal (i.e., qualitative) and the issue of interest is
attitudinal. There also exists a lesser bias in
favor of the last alternative under these same conditions. For
example, candidates in an election in
which there were several people running for the same office
should most prefer to be listed first on
the ballot. Their second choice would be to be listed last. One
way to avoid order bias for these
types of questions is to rotate the response categories as one

conducts the survey (that is, each
alternative is listed an equal number of times in each position).
When, instead, the question of interest asks for some type of
numerical estimate, order bias has the
opposite effect. Subjects will tend to select a response from the
middle of the rank ordered
numerical categories (that is, there is a bias in favor of the
middle alternatives). This bias exists
because respondents often assume that the more extreme
numerical values are less likely to occur.
This problem can be partially overcome by including a wide
range of numerical categories (i.e., the
end categories should sufficiently extreme that few respondents
would be likely to give answers that
fall in these categories if they were responding to an open-
ended question rather than a multiple
choice question).
Determine Question Wording
There are several problems with question wording that
frequently occur other than those associated
with using ambiguous words, unfamiliar words, and/or response
categories that are either not
mutually exclusive or not collectively exhaustive. Many these
are discussed in a classic book by
Payne called "The Art of Asking Questions." Several of these
problems are illustrated and discussed
below.
Implied alternative questions
A implied alternative question is one in which one of the
alternatives is not stated in the

question, but rather only implied by the question. This type of
problem most frequently occurs with
dichotomous questions (questions for which a "yes" or a "no"
answer is expected). For example,
consider the following question:
Do you think most manufacturing companies that lay off
workers during slack periods could arrange
things to avoid layoffs and give steady work right through the
year?
Note that the alternative that "layoffs are avoidable" (which
would correspond to a "yes" response)
is stated in the question while the other alternative (i.e.,
"layoffs are not avoidable," which would
correspond to a "no" response) is not stated in the question but
rather is only implied by the
question. When this particular question was asked in a survey,
63% of respondents answered "yes,"
7
indicating that they felt "layoffs are avoidable." To "correct" an
implied alternative question one
need only state all alternatives within the question. For
example, the above implied alternative
question might be corrected by adding the phrase "or do you
feel layoffs are unavoidable" as in the
revised question below:
Do you think most manufacturing companies that lay off

workers during slack periods could arrange
things to avoid layoffs and give steady work right through the
year or do you feel layoffs are
unavoidable.
When this revised question was asked in different, but
comparable survey, the results were
dramatically different. Only 35% of the respondents in this
second survey replied that they thought
that layoffs were avoidable, a difference of 28% (i.e., 63% -
35% = 28%). Obviously, implied
alternative questions can create a strong bias in favor of the
stated alternative when the issue of
interest involves attitudes or subjective judgments, particularly
if a substantial proportion of those
sampled does not hold strong opinions. This is because many
people will merely agree with
whatever position is stated when they don't feel strongly about
an issue. However, implied
alternative questions are not typically a problem when the issue
is factual in nature (for example:
“Do you own a pet?”).
Wide open or not?
"Wide open" questions are questions which are insufficiently
precise for reasons other than using
words that are considered to be ambiguous. They fail to
precisely define the issue of interest because
they refer to a category that is much too broad. For example,
consider the following item:
What brand of soap do you like best?

The problem with this question is that it is not clear whether it
is soap for bathing, washing dishes,
washing clothes, or some other type of soap that is of interest.
One might correct this question by
merely specifying the type of soap that is relevant as was done
in the revised question below.
What brand of dishwashing soap do you like best?
Complementary alternatives
As a general rule, when dichotomous questions are used to
measure respondents' opinions on some
issue, the alternatives presented should be equally extreme.
Below is a question that violates this
principle.
Would you say it's better to regulate business pretty closely, or
would you say the less regulation of
business the better?
The alternative "the less regulation of business the better"
suggests that it is best to have no
regulation of business at all. As such, this alternative is more
extreme than the other alternative (i.e.,
"it's better to regulate business pretty closely"). Since most
people tend to avoid taking extreme

8
positions, there will be a bias in favor of the less extreme
alternative. To correct the problem one
needs only to replace the more extreme alternative with a more
moderate alternative.
Pie a la mode
A "pie a la mode" question is one that provides alternatives two
different but related issues. For
example, consider the question below:
Would you say that the mayor is doing a good job of running
the city or that he could do a better
job?
At first glance this question appears to be quite reasonable.
However, it does involve two related
issues: 1) is the mayor doing a good job or not; and 2) could the
mayor do a better job or not. It is
possible that a respondent might feel that the mayor is doing a
good job but still think that he could
do a better job than he is currently doing. (Many of us have
heard our parents say to us that we did
well in school but that we could still do better. We have also
been asked whether we wanted ice
cream or pie for dessert when we wanted both - i.e., pie a la
mode) Hence, the respondent might
agree with both alternatives. To correct "pie a la mode"
questions one needs to formulate two new
questions, one for each of the issues represented in the original
question as illustrated below.
Would you say that the mayor is doing a good or a poor job of

running the city?
Do you think the mayor could do a better job of running the city
or that he is not capable of doing
better?
Note: When correcting a pie a la mode question, it is important
that you not create two implied
alternative questions.
Balance
Some questions are "unbalanced" in their alternatives in the
sense that some "ideas" are represented
more than once in the alternatives. For example, consider the
following question:
Which one of these things most influenced your decision when
you most recently purchased a hat?
____ style
____ good looks
____ workmanship
____ material
____ appearance
The problem with this question is that "how the hat looks" is
represented three times in the set of
alternatives (i.e., style, good looks, and appearance all relate to
how the hat looks and mean much
the same thing). When an idea is represented multiple times in a

set of alternatives it tends to be
chosen more often. Therefore, to correct a question of this type
one need only delete all but one
reference to any given concept. In the case of the above
question, one might want to delete "good
looks" and "appearance" and retain "style."
9
Double-barreled questions
Double-barreled questions involve alternatives which are
combinations of positions to two different
issues. Furthermore, the list of alternatives often does not
include all possible combinations. For
example, consider the following item:
Do you prefer to date people who are short with dark hair, or
tall with blonde hair?
Obviously, some people might prefer short and blonde, or tall
and dark, the two missing
alternatives. Hence, the respondent may not want to select
either of the two alternatives. One could
either list these two missing alternatives or, preferably, create
two separate questions (assuming, of
course, that the issues are independent). For example, the above
double-barreled question might be
corrected by splitting it into two questions in the following
manner.

Do you prefer to date people who are short or tall?
Do you prefer to date people who have dark or blonde hair?
How specific?
These questions require the respondent to estimate a percentage
but fail to specify the base on
which the percentage is to be calculated even though many
bases are possible. For example, consider
the following item.
About what percent of after tax profit would you say that your
company made last year?
There are many different bases which one could use to calculate
a profit percentage (for example:
gross sales, net sales, stockholders' equity, assets, etc.). These
different bases would provide very
different percentages for a given level of profit measured in
dollars. One might correct this problem
as follows:
About how much after tax profit as a percent of sales would you
say that your firm made last year?
Extensive questions
Extensive questions are questions that require respondents to

engage in a considerable amount of
mental calculations to arrive at a reasonably accurate answer.
For example, consider the following
item.
How many eggs do you fry in a year?
Many people will not have a good idea of how many eggs they
fry in a given year stored in their
memories. It is something to which most people give little, if
any, thought. As such, they must
develop an estimate. They could try to recall individual
instances of frying eggs and add these
together. However, this would obviously take a very long time
and people would not recall many of
these instances. Probably the best way to develop such an
estimate would be to think of how many
eggs they fry in a typical week and multiply this figure by 52.
However, not everyone will think of
this method. A good way to overcome this problem would be to
shorten the time period over which
the estimate is to be made as in the revised question below.
10
About how many eggs do you fry in a typical week?
If the researcher wants an estimate for an entire year, he/she can
multiply the respondent's reply by
52. This could be done by computer and would be much more

accurate than asking the respondents
to make these calculations.
Uncomplimentary exceptions
These questions involve exceptions to the general rule that the
alternatives to dichotomous
questions should be equally extreme. For example, consider the
following items:
Do you think salaries of teachers like yourself are too low or
too high?
Few individuals are likely to think they are overpaid and even
less likely to admit that they might be
overpaid even if they felt that way. Hence, given the above
question, almost the entire sample would
be "forced" to say that their salaries were too low even though
many might think their salaries were
about right. To correct this problem one could change the "too
high" alternative to "about right" as
indicated in the revised question below.
Do you think salaries of teachers like yourself are too low or
about right?
Determine question sequence
Information types and order of collection
There are three types of information that can be collected on a

questionnaire: 1) basic information;
2) classification information; and 3) identification information.
Basic information is the
critical information that relates directly to the research project –
the questions that answer the key
information needs of the survey (for example, capturing
respondents opinions and attitudes about
the product or service of interest.) Classification information
is the socioeconomic, demographic
and lifestyle characteristics (for example, age, income, gender,
marital status, occupation, number of
children in household, etc.) of respondents that are used to
classify respondents and interpret their
answers. Identification information includes respondents’
names, addresses, telephone numbers.
Identification information does not have to be collected on a
survey – in fact, it rarely is because
researchers usually want to offer respondents the opportunity to
answer freely by remaining
anonymous. Identification information may be collected for a
few legitimate reasons. For example,
identification information may be needed if field service
supervisors need to verify that respondents
were actually interviewed, or to remit promised incentives to
respondents for participating, or if
follow-up contacts are needed to collect additional data and link
it to the information previously
given by a specific respondent (for example, to measure changes
in brand attitudes over time.)
As a general guideline, basic information should always be
obtained first in a
questionnaire, followed by classification information, and
finally, identification information
if it is necessary. The basic information is of the greatest
importance to the research study and

should be obtained first, before we risk alienating respondents
by asking a series of personal
11
(classification information) questions. Classification
information is often boring to respondents. In
addition, classification questions often pertain to information
that respondents may consider private,
so they may be sensitive about answering them before they have
become involved and engaged in
the questionnaire. As a result, if these questions come first,
many respondents may decide that they
don't want to participate in the survey and terminate the
interview or throw away the questionnaire.
When the classification questions are placed at the end of the
questionnaire, the respondent will
have already made an investment in the survey by answering the
"product" oriented questions and
will usually complete the interview or questionnaire. Therefore,
never ask for demographic
information at the beginning of a questionnaire.
Opening Questions
The opening question, and in some cases the opening set of
questions, should be designed to win
the respondents' interest in the survey. The question should be
easy to answer and should relate to
something about which the respondents would like to express
their opinions or perhaps would enjoy
relating their experiences. If telephone or personal interviews

are being conducted, an open-ended
question will often get the respondent talking and involved in
the survey. For example, if you were
conducting a personal interview survey for a restaurant, you
might want to open your interview with
one of the following questions.
How do you feel about the availability of good restaurants on
Mill Ave.?
Have you recently had any particularly enjoyable experiences at
restaurants on Mill Ave.?
In contrast, a written questionnaire should never open with an
unstructured question. However the
structured opening question should still meet the criteria of
being interesting and easy to answer.
Difficult questions
Place the difficult questions in the middle of the questionnaire
so that the respondent will have had
some time to think about the general topic area before being
challenged with these items and will be
able to give more thoughtful answers. Furthermore, if difficult
questions are asked early in the
questionnaire, the respondent may feel the study is too difficult
for them and will terminate the
interview.
Funneling
The sequencing of questions within sections of the
questionnaire is often important because earlier

questions can sometimes bias the responses to questions that
follow them. Consider the bias that is
likely to occur given the order of the following two questions:
Do you ever watch “Survivor”?
What are your favorite television programs?
The second question is likely to elicit many more references to
Survivor when the questions are
asked in this sequence than if their order were reversed. Not
only does the first question make the
mentioned program more salient to the respondent, but it also
suggests that the sponsors of the
study have a special interest in this program. However, asking
about the subjects' favorite television
12
programs first is not likely to lead the respondent. If the
specific program names must be addressed
in a survey, they should appear in the following order.
What are your favorite television programs?
Do you ever watch “Survivor”?
This procedure is referred to as “funneling” – that is, starting
broad and getting more specific as
the questionnaire continues, just as the image of the funnel
suggests. Funneling is particularly
necessary when brand perceptions are being measured.

Therefore, as a general rule, don't ask
questions that identify the sponsor of the survey until it is
absolutely necessary to do so.
Arrange the questions in a logical order (for example, how they
first became aware of the store, what
prompted them to shop there the first time, what they like and
dislike about the store) and group the
questions by topic (for example, how various brands rate on
different product attributes). This will
make it easier for the respondents to complete the questionnaire
and they will be likely to provide
more accurate answers.
Layout and reproduction of the questionnaire
The layout and reproduction of the questionnaire is particularly
important for mail surveys for at
least three reasons. First, these factors can have a strong effect
on the response rates obtained in
mail surveys. Two major considerations in many prospective
respondents’ decisions of whether to
participate in a mail survey are their perceptions of how much
effort will be required to complete the
questionnaire and their assessments of how important the study
appears to be. Poorly designed
questionnaire layouts and inferior reproduction methods will
make the questionnaire appear more
complex to complete and will create doubts about the
importance of the survey. Most respondents
will tend to believe that if the study were really important, the
sponsors would have spent the time,
effort, and money necessary to design a professional looking
questionnaire.
Second, if the layout or reproduction of the questionnaire is

inadequate in some respect, it is likely
that many of the respondents will make errors in recording their
answers. For example, some
respondents may not be able to follow the skip patterns in the
questionnaire and respond to the
wrong set of questions. Others may check the wrong response
categories when these categories are
placed close together. However, online survey software (e.g.,
Qualtrics) has eliminated this problem
at large (e.g., you can set up skip logic using Qualtrics).
Finally, a poorly constructed questionnaire can cause
difficulties in coding and entering the data. For
example, when the questionnaire is not precoded (that is,
numerical values are not pre-assigned to
the response categories and printed on the questionnaire), this
function will have to be performed
manually at greater expense and with a higher error rate.
Furthermore, when response categories are
placed close together, it is likely that those entering the data
will enter the wrong values. Again,
online survey software (e.g., Qualtrics) has elimated this
problem at large (e.g., you no longer enter
data manually as Qualtrics records the data automatically).
Increased errors in recording the raw data and entering it into
the computer for data analysis can
also occur in telephone and personal interviews. Furthermore,
inadequate layout and reproduction
of the questionnaires can increase interviewing time, and,
hence, the cost of the survey.
13

There are some generally accepted rules that one should keep in
mind when designing the layout
and deciding on the reproduction of a questionnaire:
Precode all multiple choice and dichotomous questions.
Avoid making the questionnaire pages appear crowded. There
should be a lot of "white
space" on every page.
Separate the response categories from the question itself. Leave
enough room in each
category to comfortably respond. Cramped response categories
encourage non-response.
Make response categories for multiple choice questions vertical
whenever possible.
Horizontal groupings of response categories are more difficult
to read.
When the same response categories (e.g., "yes" and "no", or
“strongly disagree/strongly
agree) are used for a group of questions placed together in the
questionnaire, arrange the
questions and the response "boxes" to be checked (or numbers
to be circled) so that the
response categories appear as column headings over the
responses boxes or numbers.

For example:
Indicate whether you own each of the following kitchen
appliances.
Yes No
dishwasher ____ ____
trash compactor ____ ____
electric knife ____ ____
Pretesting the questionnaire
Most questionnaire problems can be caught in a properly
conducted pretest. Unfortunately, some
questionnaires are never pretested, and even when pretests are
conducted, the procedures used are
often woefully inadequate. Too often, those designing
questionnaires feel that because they have no
trouble understanding the questions, their intended respondents
will have no difficulty
comprehending these same questions. However, the researchers
knew what the question was
intended to them.
An ideal pretest of a questionnaire would involve the following
steps:

Match each question with the segment of the statement of
information needs to which it
relates. If this is difficult to do, the statement of information
needs is inadequate and
should be revised before proceeding.
14
Evaluate whether each individual question accurately reflects
the specific information
required. If this cannot be easily determined, the statement of
information needs is
probably insufficiently specific and should be revised before
proceeding.
Those who constructed the questionnaire should try to complete
the questionnaire as if they
were a respondent and someone else had prepared the
questionnaire. In this regard, it is important
that they read each question as if they did not already know
what it was asking. They should ask
themselves whether any of the questions or response categories
could possibly be interpreted in
more than one way. If more than one reasonable interpretation
is possible, the item should be
revised. Next, they should try to provide an accurate response to
each question and assess whether
they have any undue difficulty in doing so. In some cases the

researchers will not be members of the
population to be sampled. When this is the situation, they must
try to identify with some member of
the population of interest and try to formulate answers using the
same thought patterns that they
would expect their "referent" to employ. If the researchers have
difficulty doing this, the researchers
either have not done a sufficient amount of exploratory research
or there are problems with the
question and it should be revised. It is possible that the
respondents will not be able to provide
accurate answers to the exact questions of interest to the
researchers. When this is the case, the
researchers must revise their statement of the information to be
collected along with the question
and accept something less than they would ideally like to have.
Critical feedback from an experienced survey researcher who
did not participate in designing
the in questionnaire should be sought at this stage. A person of
this type can often catch
many of the most serious remaining problems and identify other
potential problem areas for
investigation in field pretests of the questionnaire.
The initial field pretest of the questionnaire should involve
personal interviews regardless of
whether the survey is to be conducted by mail, telephone,
Internet, or through personal
interviews. A small convenience sample from the population of
interest would be acceptable
at this stage. It is extremely important that the researchers
assure the subjects participating in
this phase that the purpose of the interview is to detect
difficulties with the questionnaire so

that these problems can be corrected before the major survey is
conducted. Basically, the
researcher is asking these might react to the questionnaire.
Depersonalizing the task in this
manner makes it easier for the respondent to be critical. They
don't have to be concerned
about insulting the researcher or admitting that there is
something in the questionnaire that
they don't personally understand (i.e., I understand this
question, but I think there are other
consumers who wouldn't). These subjects should also be
informed of the types of problems
that might exist, including: 1) there might be more than one way
to interpret some of the
questions and/or response categories and it is possible that some
subjects may have little, if
any, idea what other questions mean, 2) some of the words used
in the questionnaire may
not be familiar to everyone in the population of interest, 3) the
response categories for some
questions may be inadequate (i.e., not mutually exclusive and
collectively exhaustive), 4)
some subjects might consider some of the questions to be
insulting or otherwise irritating, 5)
some of the questions may not be relevant for everyone and
more skip patterns are needed,
6) some of the questions may ask for information that is very
difficult, if not impossible for
the subjects to provide, 7) some of the questions might ask for
information that is overly
personal or otherwise sensitive, 8) perhaps the questionnaire is
uninteresting, and 9) the
questionnaire could be too long.
Unfortunately, many researchers conduct a pretest by merely

handing out questionnaires and
seeing whether people will mark a response category for each
question. Sometimes they will
15
tell the subjects that they are conducting a pretest and they want
to find out if anything is
wrong with the questionnaire. However, this rarely results in
any useful information about
what could be wrong with the questionnaire. People will mark
response categories to the
most insane questions. Furthermore, they are unlikely to
mention that anything is wrong
unless they are convinced that the researcher is very interested
in getting critical feedback.
Subjects don't like to tell researchers that there are mistakes in
their questionnaires for fear
of insulting them, and subjects hate to admit to researchers that
there is something in the
questionnaire that they don't understand because it might make
them look stupid.
After revising the questionnaire to eliminate the problems
discovered in the initial field
pretest, a second field pretest should be conducted using a quota
sample representing the full
range of the population. For example, if the general population
is to be surveyed, this pretest
should included subjects from the lower, middle and upper
social classes. The same
interviewing procedures would be used as those employed in the
initial field pretest. When

this pretest necessitates substantial revisions in the
questionnaire, it should be repeated.
When economically feasible, it is desirable to conduct a final
pretest under the same field
conditions to be used in the major survey (i.e., same method of
communication; incentives,
if any; etc.). No probing of the subjects regarding problems
with the questionnaire would be
involved. The purpose of this final pretest would be to estimate
such things as the response
rate, the percent of incomplete questionnaires, and the cost per
interview. This step is only
reasonable for very large surveys.
It is not infrequently the case that constructing the
questionnaire of interest involved making only
minor modifications to a former questionnaire that had been
successfully used to survey a
comparable sample of respondents. When this is the case, only
the changes need to be thoroughly
tested.
Discussion – Questionnaire Items
(Read Chapter 13 and related handouts first)
1. Following are six questions that might be found on
questionnaires. Comment on each as
to whether or not it is a good question. If it is not, explain why.
(Assume that no lead-in

or screening questions are required. Judge each question on its
own merits.) Be critical
and specific!
a. Do you read the Wall Street Journal regularly?
b. Why did you decide to attend Greendale University?
c. When did you first start chewing gum?
d. What percentage of your time is spent asking for information
from others in your
organization?
e. How much discretionary buying power do you have each
year?
f. Do you think the president is doing a good job now?
2. In a class project, students developed a brief self-
administered questionnaire by which
they might quickly evaluate a professor. One student submitted
the following instrument.
Evaluate the questions asked and the format of the instrument.
(Note: this is a paper-
and-pencil survey and below is the EXACT instrument used in
the survey. Make sure you
evaluate the content of each question, the order of questions,
layout/format of the

questionnaire, etc.) Be critical and specific!
Professor Evaluation Form
1. Overall, how would you rate this professor?
Good Fair Poor
2. Does this professor
a. Have good class delivery? _____
b. Know the subject? _____
c. Have a positive attitude toward the subject? _____
d. Grade fairly? _____
e. Have a sense of humor? _____
f. Use audiovisuals, case examples, or other class room aids?
_____
g. Return exams promptly? _____
3. What is the professor’s strongest point?
4. What is the professor’s weakest point?
5. What kind of class does the professor teach?
6. Is this course required?
7. Would you take another course from this professor?

TIPSHEET – SENSITIVE QUESTIONS
Researchers sometimes ask sensitive questions in surveys.
Respondents are often hesitant to
answer sensitive items, so item nonresponse on these questions
is normally higher than for other
questions in a survey. Some respondents may even stop taking
the survey because a sensitive
question turns them off from the process. This tipsheet deals
with two types of sensitivity: social
desirability and privacy.
A common source of bias in surveys is social desirability bias.
This refers to the tendency of
respondents to overreport socially favorable attitudes and
behaviors on sensitive questions. For
example, it is socially undesirable to hold racist or sexist
attitudes. Many respondents who hold
these attitudes realize that their opinions are considered “bad”
by social norms, so they are
hesitant to admit them even in the relatively anonymous setting
of a survey. Such respondents

will report more egalitarian beliefs than they really hold.
The same logic applies to behaviors. Wearing a seatbelt while
driving is a “good” behavior, but
many people do not do it. Respondents may tend to overreport
seatbelt usage, causing the survey
to overestimate rates of seatbelt wearing.
Sometimes social desirability is not a problem with a question,
but respondents may still be
uncomfortable answering it because they are concerned with
privacy. This is especially the case
with questions about politics, religion, and demographics. Many
people cling to the belief that
politics and religion are not spoken about in polite company, so
they may be uncomfortable even
in the survey setting disclosing for whom they voted or what
church they attend. Even basic
demographic questions such as race, sex, age, and income are
often met with a “that’s none of
your business” attitude from respondents.
Sensitivity is difficult to combat, but there are some techniques
survey researchers can use to

reduce bias that results from it:
Anonymity and Confidentiality
Always reassure respondents about their anonymity or
confidentiality in the introduction to the
survey. Remind them of these assurances later in the survey
when introducing sensitive
questions. Researchers may even want to state explicitly that no
one (outside of the research
team) will ever be able to match respondents’ identities to their
answers. For demographic
questions, it sometimes helps to say that these questions are
asked for analysis purposes only.
Respondents may be put at ease the more researchers can
reassure them of their privacy, so
repeat these reassurances as often as needed. Confidentiality
Avoid putting sensitive questions too early or too late in the
survey.

It is generally not a good idea to start the survey with any
question that touches on something
private. When respondents start a survey, they are generally not
drawn into the process yet or
committed to finishing it. Sometimes respondents start a survey
to see if the first few questions
are interesting, then decide whether it is worth finishing it.
Putting a sensitive question up front
immediately raises a red flag with respondents who have
privacy concerns and increases the
likelihood that they will break off the survey. It is better to lead
the questionnaire with simple
items that draw respondents into the survey process and engage
their interest. If there are no
other viable alternatives, it is acceptable to start the survey with
simple demographics, but this
approach is not ideal. Never put a question with social
desirability concerns first.
Placing sensitive items at the end of the survey is not a great
idea either. Dealing with sensitive
questions can be unpleasant for many respondents, even if they
choose not to answer them.
Researchers should not risk ending the survey with respondents

feeling suspicious or offended,
especially if the research plan involves recontacting respondents
in the future. Generally, it is
acceptable to end the survey with demographics as these are
usually the least important items
and raise the lowest level of privacy concerns, but never end a
survey with questions that raise
social desirability issues or that ask about actions or attitudes
that might be considered private.
Sensitive questions should be placed around the middle of the
survey, usually half way to two-
thirds of the way through the questionnaire. Putting them here
lets the researcher draw the
respondent into the response process and build a base level of
trust before sensitive topics are
raised. This strategy also leaves room for less sensitive
questions to be posed closer to the end of
the questionnaire. Again, if you must start or end the survey
with demographics, that approach is
acceptable. However, always put the most sensitive items closer
to the middle.

Make respondents feel comfortable telling the truth.
Do you support or oppose drawing school attendance zones to
make schools more
racially diverse?
The question above is a simple one, but it raises concerns with
social desirability. Respondents
may feel pressured to answer that they support this type of
zoning plan. If respondents oppose
the idea because they hold racist beliefs or are uncomfortable
with racial diversity, they may feel
that giving an “oppose” answer reveals their true attitude to the
interviewer and makes them look
racist. Thus, they answer “support” instead to avoid revealing a
socially undesirable attitude.
Some respondents may oppose diversity-based districting plans
for reasons that have nothing to
do with race. Perhaps they know that these plans often involve
bussing students far across a city
in order to diversify schools, and would prefer that students
(including their own, maybe) attend

a neighborhood school that is closer to home. These respondents
may also feel pressured to
answer “support” because they fear that an “oppose” answer
makes them look racist, even
though their attitude has nothing to do with race per se.
The researcher wants to make respondents feel comfortable
enough to reveal their true attitudes,
so the question can be reworded or introduced in a way that
makes an “oppose” answer look
more acceptable. This might be one alternative wording:
Race is a topic that makes many people uncomfortable, but the
government often makes
policies that deal with racial issues. We’d like to know how you
feel about some of these
policies. There is no right or wrong answer, and you can choose
not to answer the
question for any reason. Do you support or oppose drawing
school attendance zones to
make schools more racially diverse?
The same approach also works with behaviors. The first
question below is a straightforward item

about whether the respondent voted in the last election. Many
people lie about voting, however.
“Good citizens” are supposed to vote, so respondents are often
wary of admitting that they
skipped an election. This is one reason why political surveys
tend to significantly overreport
voting rates. The second question below frames the voting
question in a way that signals to the
respondent that not voting is an acceptable response.
Did you vote in the November election?
In talking to people about elections, we find that a lot of people
were not able to vote
because they weren’t registered, they were sick, or they just
didn’t have the time. How
about you? Did you vote in the election this November?
Mode matters!
The mode in which a survey is deployed significantly affects
how sensitivity shapes the results.
Respondents are more hesitant both to answer sensitive

questions in the first place and to answer
them truthfully in modes where a human interviewer is present.
Thus, social desirability bias and
privacy concerns are bigger issues in face-to-face and phone
surveys. Respondents feel less
anonymous in these formats and prefer to project a positive
image to the interviewer, so they are
less willing to disclose sensitive information.
However, surveys administered without a human interviewer
reduce sensitivity effects
substantially. Respondents are more willing to disclose private
and socially undesirable
information about themselves in web and mail surveys where
there is not the pressure of
maintaining a positive image in front of an actual human being.
Self-administered modes do not
eliminate social desirability and privacy concerns, but
researchers who are especially interested
in asking about sensitive topics should give serious
consideration to the merits of using them (or
even a mixed-mode design). Self-administered surveys overall
tend to yield lower data quality
and lower response rates, but these tradeoffs may be worth

greater disclosure on sensitive items.
Author: Patrick R. Miller, DISM Survey Research Associate

Provide a 200 word or more discussion on a topic related to the EP.docx

Recommended

Recommended

More Related Content

Similar to Provide a 200 word or more discussion on a topic related to the EP.docx

Similar to Provide a 200 word or more discussion on a topic related to the EP.docx (20)

More from briancrawford30935

More from briancrawford30935 (20)

Recently uploaded

Recently uploaded (20)

Provide a 200 word or more discussion on a topic related to the EP.docx