Floyd Jackson Fowler analyzed survey data where unclear terms in questions led to biased results. He found 7 health survey questions with poorly defined key terms that respondents inconsistently understood. When the questions were revised to clarify the unclear terms, the estimates from the surveys significantly changed, showing that ambiguous questions can produce inaccurate data. Systematically pretesting questions to identify unclear terms and refining question wording can reduce errors in survey measurement.

1. American Association for Public Opinion Research
How Unclear Terms Affect Survey Data
Author(s): Floyd Jackson Fowler, Jr.
Source: The Public Opinion Quarterly, Vol. 56, No. 2 (Summer, 1992), pp. 218-231
Published by: Oxford University Press on behalf of the American Association for Public Opinion Research
Stable URL: http://www.jstor.org/stable/2749171 .
Accessed: 28/09/2011 16:21
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
American Association for Public Opinion Research and Oxford University Press are collaborating with JSTOR
to digitize, preserve and extend access to The Public Opinion Quarterly.
http://www.jstor.org

2. HOW UNCLEAR TERMS AFFECT
SURVEY DATA
FLOYDJACKSONFOWLER,JR.
Abstract Although writing clear questions is accepted as a gen-
eral goal in surveys, procedures to ensure that each key term is
consistently understood are not routine. Researchers who do not
adequately test respondent understanding of questions must as-
sume that ambiguity will not have a large or systematic effect on
their results. Seven questions that were drawn from questions
used in national health surveys were subjected to special pretest
procedures and found to contain one or more poorly defined
terms. When the questions were revised to clarify the definition
of key terms, significantly different estimates resulted. The impli-
cation is that unclear terms are likely to produce biased esti-
mates. The results indicate that evaluation of survey questions
to identify key terms that are not consistently understood and
defining unclear terms are ways to reduce systematic error in
survey measurement.
Introduction
It has been axiomatic at least since Stanley Payne in 1951 wrote his
classic book, The Art of Asking Questions, that survey questions
should be clear. Ideally, they should mean the same thing to all respon-
dents, and they should mean the same thing to respondents as to the
researcher, the person who will interpret the answers.
Although the principle seems clear, putting this guideline into prac-
tice may be more difficult. The main step that most researchers take
Public Opinion Quarterly Volume 56 218-231 ? 1992 by the American Association for Public Opinion Research
All rights reserved. 0033-362X/92/5602-0003$02.50
FLOYD J. FOWLER, JR., iS senior research fellow at the Center for Survey Research,
University of Massachusetts-Boston. Some of the data in this article were presented
at the Fifth Conference on Health Survey Research Methods, Keystone, Colo., 1989.
This project was supported by grant HS 05616 from the National Center for Health
Services Research and Health Care Technology Assessment. This analysis emerged
from a research project that, in addition to the author, was conducted by Charles F.
Cannell, Graham Kalton, Lois Oksenberg, and Katherine Bischoping at the Survey
Research Center, University of Michigan.

3. Unclear Questions 219
to evaluatetheir questionsis the pretest. The most commonstrategy
for pretesting survey instrumentsis probablyto have a few experi-
enced interviewers take 20-40 interviews using a pretest interview
scheduleandthen participatein a debriefingmeetingwith the investi-
gators. Although there have been protocols outlined for pretesting
(ConverseandPresser1986;DeMaio1983),even attheirbest, pretests
tend to be subjective and unsystematic. The criteriafor what inter-
viewers are lookingfor andthe standardsthey shoulduse for when a
"problem"exists areseldommadeexplicit.Eachinterviewerhasonly
a smallsampleof respondents,so the base for evaluatingquestionsis
small. Indeed, the pretestputs the interviewerin a complexdualrole
of interviewerand observer, andit is not clearhow well interviewers
can play both roles simultaneously.To quote Presser (1989, p. 35):
"Althoughit is likely that pretested questionsare betterthanunpre-
tested questions,thereis ever-mountingevidencethatmanypretested
items performpoorly."
Muchof thatevidence resultsfromeffortsto involve cognitivepsy-
chologistsin surveyresearchmethodology(Jabineet al. 1984;Lessler
1987).When standardsurvey questionsare examinedin a laboratory
context, ambiguitiesfrequentlyare uncovered(Royston 1989;Willis,
Royston, andBercini1989).Debriefingof respondentsina fieldsetting
also routinely reveals inconsistent understandingof terms (Belson
1981).
Ambiguousterms pose problemsfor interviewersin surveys. The
rules of standardizationlimit the extent to which they can or should
provide clarificationof unclear terms (Fowler and Mangione 1990).
Thestandardizedresponseinterviewersaretaughtthatsaying,"What-
ever it means to you," may be appropriatefor terms that describe
subjective states-such as "problems," "satisfaction," or "happy"
-but this seems inadequateforquestionsaboutfactualeventsorexpe-
riences such as "visits to doctors" or "havingarthritis."
The most importantissue, however, is the way questionambiguity
affectsthe qualityof surveydata.Therearetwo ways thatambiguous
terms can affect the resultingdata. If the effect of ambiguityon an-
swersis unsystematic,thereis moreerrorin the measurementandthe
abilityto reachvalid conclusionsaboutrelationshipswill be reduced,
but the estimates of means and the overall distributionsmay not be
affected. If the ambiguitysystematicallyleads respondentsto err in
one direction, by definition,estimates based on the answers will be
biased. Althoughneitherkind of erroris desirable,most researchers
probablywouldbe particularlyconcernedaboutunclearquestionsthat
producedbiased estimates-estimates that consistentlydifferedfrom
the true value of what they were tryingto measure.

4. 220 Floyd Jackson Fowler, Jr.
This paper emerged from my efforts to develop more systematic
ways to use pretests to identifyproblemsurvey questions, including
those that are not understoodconsistentlyby respondents.As will be
describedbelow, buildingon the work of Cannell(Cannell,Fowler,
and Marquis 1968; Cannell, Lawson, and Hausser 1975; Cannell
and Oksenberg 1988) and Morton-Williams(Morton-Williams1979;
Morton-Williamsand Sykes 1984),the behaviorsof interviewersand
respondentsduringinterviewswere coded for clues to questionprob-
lems. Among the question problems identifiedwere some unclear
terms or concepts. In those cases, the questionswere revised to im-
provethe definitionof key termsandthentestedagain,usingcompara-
ble samplesandprocedures.Hence, thatresearchprovidedan oppor-
tunityto look at the effect of havinguncleartermsin questionson the
estimatesthat emergefrom surveys.
Method
Thedatafor this papercome froma studydesignedto evaluatestrate-
gies for pretestingsurvey questions.A 60-itemsurveyinstrumentwas
constructed, using questions drawnfrom health survey instruments
conductedby governmentagenciesor academicsurveyorganizations.
Althougha true samplingof such survey items proved impossible,
questionswere chosen thatwerejudgedto be representativeof ques-
tions commonly used in health surveys of the generalpublic. Items
were chosen to cover the rangeof thingstypicallymeasuredin health
surveys, includingutilization of medical care, health status, health
beliefs, andbehaviorsrelatedto health.
Samplesfor all the testingdescribedin this paperwere drawnfrom
telephone subscribersin southeasternMichigan. Since the purpose
was to evaluatepretestingproceduresandnotto makegeneralpopula-
tion estimates, strictprobabilitysamplingprocedureswere not used.
Interviewerswere permittedto substitutea new numberafter three
unsuccessful attempts to contact a household or when someone re-
fused to be interviewed.The protocolfor respondentselectionwithin
a householdcalledfor selectingamongthose who were at homeat the
time of contact. However, the proceduresfor contactinghouseholds
andsubstitutingandchoosingrespondentswerethe sameforallphases
of the projectso thatresultsfromthe varioustests couldbe combined
andcompared.
Inthe initialphaseof the study,the test healthinterviewsurveywas
usedfor 110pretestinterviews.Sixty of these weredoneby interview-
ersfromthe Universityof Michigan'sSurveyResearchCenterand50,

5. Unclear Questions 221
by interviewersfromthe CenterforSurveyResearchatthe University
of Massachusetts-Boston. All interviews were tape-recorded,with
respondentpermission,of course.
The behaviors duringthe pretest interviews were then coded by
usinga scheme adaptedfromthatreportedby CannellandOksenberg
(Cannell,Lawson, and Hausser 1975;Cannelland Oksenberg1988).
The purposeof the codingwas to identifyquestionsthatappearedto
pose problemsfor interviewersor respondentsthat were reflectedin
theirbehaviorsduringthe interviews.The behaviorsthatprovedto be
particularlycriticalwere the rates at which respondentsasked inter-
viewers for clarificationof questions and the rates at which respon-
dents gave at least one inadequateanswerto a question.When either
of these behaviorsoccurredin at least 15percentof the pretestinter-
views, the questionwas consideredlikely to have a wordingproblem
thatdeserved attention.
Numerouskindsof problemswereidentifiedthroughthisprocedure,
as is discussed elsewhere (Cannellet al. 1989;Fowler 1989;Oksen-
berg, Cannell,andKalton 1991).The focus of this paperis that set of
questions that seemed to pose a problemfor respondentsprimarily
because a key termor concept in the questionwas not well defined.
Ofthe 60 questions,therewere seven thatseemedto be problematic
primarilybecause of a poorly definedterm or concept. These seven
questionswere revised in an attemptto solve the definitionalproblem
andcommunicatea consistentunderstandingof thetermsto allrespon-
dents. In the revisions, the question objectives were not purposely
changed,thoughin some cases they hadto be refined.Table1presents
the exact wordingof the originalquestionsandthe revisions. In five
of the seven cases, the revision primarilyconsisted of addingwords
to the question to clarify some aspect of what was intended. In the
case of the questionaboutvisits to doctorsat an HMO,a singleques-
tion was turnedinto two questions. In the case of the questionabout
the numberof visits to doctorsin the past year, the definitionof what
does anddoes not countas a doctorvisit provedto be so complicated
that a four-question series was substitutedfor the single question.
However, the objectives andwhatwas to be measuredremainedbasi-
cally unchanged.
These revised questionsbecamepartof a revisedsurveyinstrument
thatwas used in a second roundof pretesting.In the secondphase of
pretesting, 150 respondentswere interviewed,drawnfrom the same
sampleframe as the firstphase, using identicalrespondentselection
procedures.Interviewsagainwere splitbetweenthe staffsof the Uni-
versity of Massachusettsand the Universityof Michigan,with those
staffstaking50and100interviews,respectively.Forthe secondround,

6. 222 Floyd Jackson Fowler, Jr.
60 interviews were tape-recorded, and the behaviors during the inter-
view were coded to evaluate the impact of the changes on the behavior
of interviewers and respondents.
The focus of the analysis presented below is the extent to which the
revised questions, designed to clarify ambiguous terms, also changed
the distribution of responses.
Table 1. Original and Revised Questions Designed to Clarify
Unclear Terms
Original Question Problem Revision
1. During the past 4 What counts as ex- In the past 4 weeks,
weeks, beginning ercise? beginning Monday
Monday (DATE 4 (DATE 4 WEEKS
WEEKS AGO) and AGO) and ending
ending this past this past Sunday
Sunday (DATE 4 (LATE LAST SUN-
WEEKS AGO), DAY), have you
have you done any played any sports,
exercise, sports, or done any hobbies in-
physically active volving physical ac-
hobbies? tivity, or done any
exercise, including
walking?
2. Do you exercise or What counts as ex- Do you do any sports
play sports regu- ercise? or hobbies involving
larly? physical activities,
or any exercise, in-
cluding walking, on
a regular basis?
3. What is the average Does margarine count The next question is
number of days as butter? just about butter.
each week you have Not including marga-
butter? rine, what is the av-
erage number of
days each week you
have butter?
4. (Following a ques- How many eggs con- On days when you eat
tion on the number stitute a serving? eggs, how many eggs
of days per week What does "a typi- do you usually have?
eggs are eaten): cal day" mean?
What is the number
of servings in a typ-
ical day?

7. Table I (Continued)
Original Question Problem Revision
5. During the past 12 What counts as a visit Have you been a pa-
months, since Janu- to doctor? tient in a hospital
ary 1, 1987, how overnight in the past
many times have 12 months, since
you seen or talked July 1, 1987?
with a doctor or as- 5a. (Not counting
sistant about your when you were in
health? Do not a hospital over-
count any time you night): During the
might have seen a past 12 months,
doctor while you since July 1, 1987,
were a patient in a how many times
hospital, but count did you actually
all other times you see any medical
actually saw or doctor about your
talked to a medical own health?
doctor of any kind 5b. During the past 12
about your health. months, since July
1, 1987, were there
any times when
you didn't actually
see the doctor, but
saw a nurse or
other medical assis-
tant working for
the doctor? (How
many times?)
5c. During the past 12
months, since July
1, 1987, did you
get any medical
advice, prescrip-
tions, or results of
tests over the tele-
phone from a med-
ical doctor, nurse,
or medical assis-
tant working for a
doctor? (How
many times?)

8. Table I (Continued)
Original Question Problem Revision
6. (Referring to the What is an HMO? Do you belong to an
place where respon- HMO or health plan
dent reported last that has a list of
seeing a doctor): people or places you
Was that place a go to, in order for
health maintenance the plan to cover
organization or your health care
health care plan costs? Was your last
(that is, a place you visit to a medical
go for all or most doctor covered by
medical care, which your health care
is paid for by a plan?
fixed monthly or an-
nual amount)?
7. During the past 12 What counts as "in The next question is
months, that is, bed more than half about extra time you
since January 1, the day"? How spe- have spent in bed
1987, about how cific does the diag- because of illness or
many days did ill- nosis have to be to injury (including
ness or injury keep count? time spent in the
you in bed more hospital). During the
than half of the past 12 months,
day? (Include days since July 1, 1987,
while an overnight on about how many
patient in a hos- days did you spend
pital.) several extra hours
in bed because you
were sick, injured,
or just not feeling
well?
NOTE.-The original questions were drawn from the core of the National Health In-
terview Survey and the Health Promotion and Disease Prevention supplement to the
NHIS, the Arizona Health Survey conducted for the Center for Health Administra-
tion, University of Chicago, by Louis Harris Associates, and a pretest version of the
Canadian Fitness Survey conducted by the Canadian Fitness and Lifestyle Institute.
In some cases, questions were modified to fit the context or flow of this interview;
two questions were adapted from a self-administered questionnaire for use in a tele-
phone survey.

9. Unclear Questions 225
Results
Table2 presentsthe distributionsof answersto theoriginalandrevised
questions. For five of the seven questions, answers to the revised
questionwere significantlydifferentfromanswersto the originalone.
The other two questionsproduceddifferencesin the predicteddirec-
tionthatdidnot quitemakethe .05standardfor statisticalsignificance.
The fact that the answers to the revised questions were different
does not necessarily mean that they were more accurate. However,
for at least six of the seven questionsthatproducedchangedresults,
there was a clear a priori reason for predictingthat the observed
changereflectedimprovedmeasurement.(a) It seems most plausible
thatthe increasedreportof exercise in two questionsstems fromthe
fact that some respondentswho walk for exercise excluded it in an-
swersto the originalquestions.(b) Specifyingthatmargarinedoes not
countas butterexplainsthedecreaseintherateof butterconsumption.
(c) The changedreportingwithrespectto eggconsumptionis certainly
more interpretable,as it is clear that some people considera serving
of eggs to be one egg, while others consider it to be two. (d) The
increasein the numberof visits to doctorsreportedin responseto the
revised series is almost certainlymorecompletethanthe reportingin
response to the single question. (e) Disabilitydays seem likely to be
underreported,so that an upwardchange is likely also to be in the
directionof greateraccuracy.The only changethatis ambiguouswith
respect to whether or not it constitutes an improvementis obtaining
medicalcare from an HMO. Althoughthe investigatorsthoughtthat
the revised question was clearer, there was not the same basis for
estimatinga directionof changethatwouldbe expectedfromthe clari-
ficationthat existed for the otherquestions.
However, the datapresentedin table3 provideadditionalevidence
aboutwhetherornotthe questionswereimproved.Allof thequestions
that were revised were classified as "problems," either because 15
percentof the originalpretestrespondentsaskedfor clarificationor 15
percent initiallygave an inadequateanswer to the question. Table 3
comparesthe rates of these two behaviorsin the originalandrevised
interviewschedules. It can be seen that, in general,rates of requests
forclarificationandratesof inadequateanswersdeclined,as one would
predict if the questions had been improved.It is particularlyworth
noting that the HMO questions showed significantimprovementon
both indicators.
The one questionthatdeterioratedfromthe perspectiveof the cod-
ingof behaviorduringthe interviewpertainedto disabilitydays. While
the markedincrease in reporteddisabilitydays was predicted,it was

10. Table 2. Responses to OriginalandRevised Questions
(Percentages)
Question Yes No Total pa
Any exercise/4 weeks:
Original 67 33 100
Revision 82 18 100 <.05
Exercise regularly:
Original 48 52 100
Revision 60 40 100 <.10
0 1 2-6 7 Total p
Number days/week have butter:
Original 33 13 29 23 100
Revision 55 14 22 9 100 <.01
1 2 3+ Total p
Number eggs (servings)/day eaten:
Original 80 15 5 100
Revision 33 61 6 100 <.01
0 1-2 3+ Total Mean p
Doctor visits/year:
Original 22 33 45 100 3.6
Revision 19 24 57 100 6.1 <.05
Yes No Total p
Last doctor visit at HMO:
Original 39 61 100
Revision 23 77 100 <.05
0 1-7 8+ Total Mean p
Disability days/year:
Original 57 36 7 100 2.6 <.10
Revision 48 33 19 100 4.0b
NOTE.-N (original) = 110; N (revised) = 150.
a
Based on two-tailed t-tests for differences between means for continuous variables and be-
tween proportions for categorical variables. Multicategory variables were dichotomized based on
the proportion in the largest category versus all others.
b Includes one person who reported 90 days, almost twice as many as the next person in either
sample. Removing that person reduces the mean to 3.4. In neither case does the difference be-
tween means meet the .05 level of significance.

11. Unclear Questions 227
Table 3. Requests for Clarification and Inadequate Answers to
Original and Revised Questions (Percentage of Interviews)
Requestsfor
Question Clarification InadequateAnswers
Any exercise/4weeks:
Original 8 17
Revision 10 13
Exerciseregularly:
Original 5* 20
Revision 0 12
Numberdays/weekhave butter:
Original 18 15
Revision 13 12
Averageservingsbutter/dayeaten:
Original 17** 7
Revision 4 17
Numberof eggs (servings)/dayeaten:
Original 33** 12**
Revision 0 0
Doctorvisits/year:
Original 30 17
Revision(4 questions)a 24 43**
Last doctorvisit at HMO:
Original 17* 27
Revision(2 questions) 2 18b
Disabilitydays/year:
Original 15 7
Revision 17 30**
NOTE. N(original) = 110; N (revised) = 60.
a Figures are the sum of behavior on the four questions.
b6 and 12 percent, respectively, for the two questions.
* p <.05 based on two-tailed t-tests for differences in proportions.
** p <.01.
not predicted that respondents would give significantly more inade-
quate answers in response to the revised question.
The series of questions regarding visits to doctors cannot be directly
compared to the original question because the revision provided four
times as many chances to give inadequate answers and to ask for
clarification as the original. In general, the data suggest that the re-
vised, more completely defined questions were also better questions
as indicated by requests for clarification and the adequacy of answers.

12. 228 Floyd Jackson Fowler, Jr.
Discussion
Rasinski(1989)and SchumanandPresser(1981)provideampledocu-
mentationthat smallchangesin questionwordingcan affectthe distri-
bution of answers. For example, Schumanand Presser found that
many more people would "not allow" Communiststo speak in the
United States thanwould "forbid"themto do so. However, most of
the comparisonsthatthey give pertainto the measurementof subjec-
tive phenomena(opinions, knowledge, and feelings) ratherthan the
reportingof factualevents. The goal in measuringsubjectivephenom-
ena is to order people. There are no right or wrong answers, and
differentdistributionsin answersto similarquestionsdo not necessar-
ily implyerrorin either.However, thatis notthecase whenmeasuring
factualphenomena.Althoughsome healthservices researchgoals in-
volve the measurementof subjectivephenomena,the preponderance
of measurementin health researchtends to be factualin nature.Re-
searcherswant to count events or they want to classify people accu-
ratelyaccordingto theirexperiencesorbehaviors.Whena respondent
asks aninterviewerwhatis meantby "a servingof eggs" orby "exer-
cise," it is not sufficientfortheinterviewerto say, "Whateveritmeans
to you." That is a nondirectiveresponse, but it does not serve the
interests of gatheringaccurate data. When the researcherinterprets
the answers,assumptionswillbe madeaboutthe wayin whichrespon-
dents interpretedthe questions.
The questions examinedhere were not the work of amateurs.All
were designedby governmentor academicsurveyexperts,andfive of
the seven questions presumablyhad been subjectedto standardpre-
tests. It is obvious in retrospect that these questions contain major
ambiguities,thoughpresumablyit was not obvious to their sponsors
and those who pretested them. The fact that in most cases revised
questions that clarifyand definekey concepts producedsignificantly
differentdistributionsof answersis compellingevidencethattherewas
considerableerrorin the responsesto the originalquestions,anderror
was almostcertainlyreducedby the revised wording.
Of course, there is a critical relationshipbetween clear question
objectives and clear questions. In fact, errorcan only be assessed in
relationshipto a clearly defined measurementgoal. One reason for
unclear questions, no doubt, is that researchershave not thought
throughexactly whatthey wantto measureorareunawareof anambi-
guity. Using the examples in the test questions, exercise could have
been definedto include or exclude walkingfor exercise; buttercon-
sumptioncould include or exclude margarine.So long as the criteria
are not specifiedin the question, however, some people will answer
the questionone way, otherswill answerit another,andtherewill be

13. Unclear Questions 229
a resultingover- or underestimateof whateverthe researcherthinks
is being counted or wants to count. One good reasonfor identifying
unclearquestionsis to alertresearchersto places wherethey need to
clarifytheirobjectives.
There is, of course, no basis for saying the revised questions are
errorfree. The question on disabilitydays provides a good example
of how a limitedrevisionof a questionmaynot addressthe totalprob-
lemposed by a question.Even thoughthis questionhas been used for
yearsintheNationalHealthInterviewSurveyandelsewhere,we think
it poses majorproblemsof several types. The revised questiontried
to clarify what being in bed for half a day due to injuryor illness
meant.In this case, we took some latitudein definingwhatseemedto
be a poorlydefinedconcept;it maystillbe poorlydefinedintherevised
question,andothersmightdisagreewithoursense of whatwas meant
by the question.To the extent thatis true,it only reinforcesthe ambi-
guity of the question for all who would try to answer it or interpret
those answers.Furthermore,we thinkthe taskgiventhe respondentis
almostimpossible:recallingnumerous,sporadic,usuallyminorevents
over a 1-yearperiod.Clarifyingthe definitionof a disabilityday should
be helpfulto understandingbut, if anything,the clarificationmayhave
made the response task harder.Some question objectives requirea
completelychangedstrategyin orderto producequestionsthatcan be
answeredaccuratelyandwith ease.
These complexitiesshouldnot obscurethe mainimplicationsof the
results. First, if one wants to have respondents provide accurate
counts of events or behaviors, all respondentsshouldbe given ade-
quatedefinitionsof what it is they are supposedto report.Moreover,
it is not acceptableto give interviewersdefinitionsto be used only if
respondentsask questions or act confused. Consistentmeasurement
requiresthatall respondentsbe exposed to the sameconsistentdefini-
tions (FowlerandMangione1990).
An obvious concern is that addingdetaileddefinitionsto questions
willmakethemlong, cumbersome,andperhapsharderforrespondents
to understand.Sometimesthatis a problem,andthebenefitsof provid-
ingdefinitionsmustbe weighedagainstthe cost. Whendefinitionsare
very complex, breakinga single question into several questions, as
was done with the questionon visits to doctors, may be the solution.
Forthose who worrythatadditionalquestionswilltakemoreinterview
time, the only response is that researchersalways have to weigh the
value of each question. However, considerthe implicitchoice: asking
a singlequestionthat respondentsdo not understandandthatwill, as
a consequence, produce biased estimates versus asking three clear
questionsthat will producemoreaccurateestimates. Certainly,there
aremanyoccasions whenthethree-questionsoptionwillbe preferable.

14. 230 Floyd Jackson Fowler, Jr.
Furthermore,there should be no occasions when a researcherdoes
not want to findout that a questionis unclearso thatthe optionscan
be explicitly considered.
To thatend, researchersneedto do a betterjob of identifyingambig-
uous termsandconcepts when they arepretestingquestions.Theold-
style pretests are not adequateto the task. The recent emphasison
laboratorytesting of questionspriorto field-pretestingthemis a laud-
able developmentthatneeds to become even morecommon.Labora-
toryworkto datehas shownthatinterviewingonlya few respondents,
even fewer than 10, can identify significantambiguitiesin question
wording. The coding of interviewerand respondentbehaviorduring
pretestsprovidesanotherway to identifyquestionsthatincludepoorly
definedterms.
Of course, requests for clarificationand inadequateanswers can
resultfrom question problemsother thanunclearterms;aftercoding
behavior,there is still a diagnostictask of isolatingthe aspect of the
questionthat is causingthe problem.However, as we have seen, the
ambiguitiesdiscussed in this paper all showed up in the coding of
interview behavior in pretests; the indicatorsof question problems
were, in most cases, reducedby the clarifications.Multiplestrategies
for identifyingunclearterms, involvinglaboratorytests andfieldpre-
tests, probablyare needed and appropriate.
The most importantmessage fromthis studyis that a commitment
to identifyinguncleartermsin questionsandprovidingadequatedefi-
nitionsis an essentialpartof good surveyresearchdesign.Whenmea-
suringobjectivephenomena,tellinginterviewersto say, "Whateverit
means to you," is not an adequatesubstitutefor writingclear, well-
definedquestions.
References
Belson, William A. 1981. The Design and Understanding of Survey Questions.
London: Gower.
Cannell, Charles F., Floyd J. Fowler, and Kent Marquis. 1968. The Influence of
Interviewer and Respondent Psychological and Behavioral Variables on the
Reporting in Household Interveiws. Vital and Health Statistics, series 2, no. 26.
Washington, D.C.: Government Printing Office.
Cannell, Charles F., Sally A. Lawson, and Doris Hausser. 1975. A Technique for
Evaluating Interviewer Performance. Ann Arbor: Survey Research Center,
University of Michigan.
Cannell, Charles F., and Lois Oksenberg. 1988. "Observation of Behaviour in
Telephone Interviewers." In Telephone Survey Methodology, ed. R. Groves et al.,
pp. 475-95. New York: Wiley.
Cannell, Charles F., Lois Oksenberg, Floyd J. Fowler, and Graham Kalton et al.
1989. "New Techniques for Pretesting Survey Questions." Final report to National

15. Unclear Questions 231
Center for Health Services Research and Health Care Technology Assessment,
grant HS 05616. Ann Arbor: Survey Research Center, University of Michigan.
Converse, Jean M., and Stanley Presser. 1986. Survey Questions: Handcrafting the
Standardized Questionnaire: Quantitative Applications in the Social Sciences.
Beverly Hills, Calif.: Sage.
DeMaio, Theresa J., ed. 1983. "Approaches to Developing Questionnaires."
Statistical Policy Working Paper 10. Washington, D.C.: Office of Management and
Budget.
Fowler, Floyd J. 1989. "Coding Behavior in Pretests to Identify Unclear Questions."
In Conference Proceedings: Health Survey Research Methods, ed. Floyd J.
Fowler, pp. 9-12. Washington, D.C.: National Center for Health Services
Research.
Fowler, Floyd J., and Thomas W. Mangione. 1990. Standardized Survey Interviewing.
Newbury Park, Calif.: Sage.
Jabine, Thomas B., Miron L. Straf, and Judith M. Tanur et al., eds. 1984. Cognitive
Aspects of Survey Methodology: Building a Bridge between Disciplines.
Washington, D.C.: National Academic Press.
Lessler, Judith T. 1987. Use of Laboratory Methods and Cognitive Science for the
Design and Testing of Questionnaires. Stockholm: Statistics Sweden.
Morton-Williams, Jean. 1979. "The Use of 'Verbal Interaction Coding' for Evaluating
a Questionnaire." Quality and Quantity 13:59-75.
Morton-Williams, Jean, and Wendy Sykes. 1984. "The Use of Interaction Coding and
Follow-up Interviews to Investigate Comprehension of Survey Questions." Journal
of the Market Research Society 26:109-27.
Oksenberg, Lois, Charles F. Cannell, and Graham Kalton. 1991. "New Strategies for
Testing Survey Questions." Journal of Official Statistics 7:349-65.
Payne, Stanley L. 1951. The Art of Asking Questions. Princeton, N.J.: Princeton
University Press.
Presser, Stanley. 1989. "Pretesting: A Neglected Aspect of Survey Research." In
Conference Proceedings: Health Survey Research Methods, ed. Floyd J. Fowler,
pp. 35-38. Washington, D.C.: National Center for Health Services Research.
Rasinski, Kenneth A. 1989. "The Effect of Question Wording on Public Support for
Government Spending." Public Opinion Quarterly 53:388-94.
Royston, Patricia N. 1989. "Using Intensive Interviews to Evaluate Questions." In
Conference Proceedings: Health Survey Research Methods, ed. Floyd J. Fowler,
pp. 3-8. Washington, D.C.: National Center for Health Services Research.
Schuman, Howard, and Stanley Presser. 1981. Questions and Answers in Attitude
Surveys. New York: Academic Press.
Willis, Gordon B., Patricia N. Royston, and Deborah Bercini. 1989. "Problems with
Survey Questions Revealed by Cognitively Based Interviews." In Proceedings,
Fifth Annual Research Conference, pp. 345-60. Washington, D.C.: Bureau of the
Census.