2. References
• Groves R.M. et al. Survey methodology, Wiley-
interscience, 2004
• Canada national statistical agency:
http://www.statcan.gc.ca/
• Eurostat methodological publications
• European Values Study (EVS)
3. INTRODUCTION :
What is a survey ?
A “survey” is a systematic method of gathering information from (a sample
of) entities for the purpose of constructing quantitative descriptors of the
attributes of the larger population of wich the entities are members
A survey usually originates when an individual or institution is confronted
with an information need and the existing data are insufficient
What is survey methodology ?
Survey methodology is the study of survey methods
4. How survey work to produce statistics
Characteristics of
the population
Inference
Characteristics of a
respondent Statistical Characteristics of
computing the sample
Inference
Respondent
answers to
quenstions
6. A survey from a process perspective
Define research objectives
Choose mode of Choose
collection sampling frame
Construct and Design and
pretest a select sample
questionnaire
Recruit and
measure sample
Code and edit data
Make postsurvey adjustments
Perform analysis
7. The life cycle of a survey from a
design perspective
Target Population
Construct
Sampling Frame
Measurement What Who
is the is the
Sample
survey survey
Response about? about?
Respondent
Edited
Response Postsurvey Adjustments
Survey statistics
The Measurement dimension describes The Representational dimention
what data are to be collected about the concerns what population are
observational units in the sample described by the survey
8. The measurement dimension
Constructs are the elements of information that are
Construct sought by the researcher :
How many incidents of crimes with victims there were in the last year;
The consumption of beer in the last month;
The degree of knowledge of mathematics of childrens…
Measurements are ways to gather information about
constructs :
Measurement Questions posed to a respondent (“During the last 6 month, did you call the police
to report something that happened to you that you thought was a crime?”)
NB: the critical task for maesurement is to design questions that produce answers
reflecting perfectly the construct we are trying to measure.
Response Response could be produced in a variety of means
But in general the nature of the response is determined by the
nature of the measurement
Editing of data may examine the full distribution of
Edited answers and look for atypical patterns of responses
Edited responses are the data from wich inference is made about the values of
Response the construct for an individual respondent
9. The representational dimension
The target population is the set of unit to be studied
Target Population The adult population living in households in 2009;
The frame population if the set of target population members
that has chance to be selected into the survey sample :
Sampling Frame In a simple case it is a list of all units in the target population, but sometimes it is a s
of units imperfectly linked to population members.
i.e. a list of telephone numbers when the target population is the adult population
Sample The sample is the group from wich measurement will be
sought. In many case it is a very small fraction of the the sampling frame
Respondents are the elements successfully measures.
Respondent Non respondents is the complement
Postsurvey adjustments consist on weighting up the
underrepresented groups in order to improve the survey
Postsurvey estimate
Because of mismatches of the sampling frame and the target population
Adjustments (coverage problems) statistics based on the respondents can differ from
caracteristics of the target population. Examination of non response patterns
may suggest an underrepresentation of some groupes relative to the sampling
frame
10. The life cycle of a survey from a
quality perspective
Construct Target Population
Coverage
Validity error
Sampling Frame
Measurement Sampling
error
Measurement
error Sample
Nonresponse
Response error
Processing Respondent
error Adjustments
error
Edited
Response Postsurvey Adjustments
Survey statistics
11. Coverage of a target
population by a frame
Undercoverage
Elements in the
target Ineligible units
population Frame population
missing from the
frame
i.e.:non telephone
household, using a Covered population Ineligible units
telephone frame to
cover the full Elements in the
household frame that are no
population member of the
Undercoverage target population
i.e.:business telephone
numbers, using a
telephone frame to
cover the full household
Target population population
12. Evaluating survey questions:
Are the answers good measures of the
intended construct?
Exemple of methods that can be used
to evaluate draft survey questions
The substantive expert review the wording, the
Expert reviews order and the structure of questions, the
response alternatives etc.
A small number of target population participate in a systematic discussion
Focus groups about the survey topic. The researcher learn about the nomenclature of the
concept, the common perspective taken by the target population on key
issues etc…
Researcher test how questions are read
Questionnaire pretest and answered. A behaviour coding is
often used
13. Evaluating survey questions:
Exemple of behavior codes for interviewer
and respondent behaviors
Interview Questioning Behavior (choose one)
1. Reads questions exacty as worded
2. Reads questions with minor changes
3. Reads questions so that meaning is altering
Respondent Behaviors (check as many as apply)
1. Interrupt question reading
2. Asks for clarification of question
3. Gives adequate answer
4. Gives answer qualifies about accuracy
5. Gives answer inadequate for questions
6. Answers “don’t know”
7. Refuses to answer
15. Traditional
data collection methods
• Mailing paper questionnaires to respondents,
who fill them out and mail them back
• Having interviewers call to respondents on the
telephone and ask them the question in a
telephone interview
• Sending the interviewers to the respondent’s
home or office to administer the questions in
face-to-face (FTF) interviews
16. Alternatives methods of data collection
OCR/ICR FAX
Optical/intelligent
caracter recognition
Computerised Self
Mail Disk by Mail E-mail Web Administered
Questionnaires
Telephone CATI TDE IVR
computer assisted Touchtone Interactive
telephone interviewing data entry voice response
Face to face CAPI computer assisted personal interviewing
SAQ
Self administered Text Audio Video
questionnaire
CASI CASI CASI
Walkman
17. Alternatives methods of data
collection (a)
OCR/ICR FAX
Optical/intelligent
caracter recognition
Mail Disk by Mail E-mail Web
18. Alternatives methods of data
collection (b)
Telephone CATI TDE IVR
computer assisted Touchtone Interactive
telephone interviewing data entry voice response
19. Alternatives methods of data
collection (c)
Face to face CAPI computer assisted personal interviewing
SAQ
Self administered Text Audio Video
questionnaire
CASI CASI CASI
Walkman
21. QUESTIONS AND ANSWERS
IN SURVEYS
A questionaire
is a standardised set of questions administered to the
respondents in a survey
Respondents are required to interpret a preestablished set
of questions and to supply the information these questions
seek.
22. The cognitive processes in
answering questions
A simple model of the survey BUT…
response process
Respondents often take shortcuts to get
Comprehension through the interview more quickly
of the question
OR
Retrieval of
information they have motives that override their
desire to provide accurate information
Judgment and
estimation Responses could be biased by
Reporting an • “acquiescence” (the tendency to
answer agree)
• “social desirability” (the tendency to
present oneself in a favourable light by
underreporting undesirable attributes and
overreporting desirable one)
23. The cognitive processes in
answering questions (2)
The “satisficing” model
(Krosnik and Alwin, 1987)
Some respondents try to “satisfy” (to take a low road answering more
superficially)
whereas others try to “optimise” (to take an high road by careful
answering questions)
Satisficing respondents do not seek to understand the question completely, just
well enough to provide a reasonable answer
24. Problems in answering
survey questions
Failure to encode the information sought
Misinterpretation of the questions
Forgetting and other memory problems
Estimation strategies
Problems in formatting answer
More or less deliberate misreporting
Failure to follow instruction
25. FORMATTING THE ANSWER
Survey items can take a variety of formats;
the most common are:
1) Open-ended qustions that call for
numerical answers
2) Closed questions with ordered response
scales
3) Closed questions with categorial
response options
26. 1 - Open-ended qustions that call
for numerical answers
Now, thinking about your physical health,
which includes physical illness and
injury, for how many days during the past
30 was your physical health not good?
Note that:
Open-ended items yield more exact information than closed items
27. 2 - Closed questions with ordered
response scales
Would you say that in general your health is:
1 Excellent
2 Very good
The interviewer is instructed to
3 Good
“please read” the answer
4 Fair categories, but not the number
5 Poor attached to them!
Note that :
• with some type of rating respondents seem to shy away from the negative end of the scale
• When the scale points have numerical labels, the label can affect the answer (e.g. if respondents are asked to rate their success in life)
28. 3 - Closed questions with categorial
response options
Are you:
1 Married
2 Divorced
3 Widowed
4 Separated
5 Never married
6 A member of an unmarried couple
Note that :
• The respondent may not wait to hear or read all the option; they may select the firs reasonable answer they consider (primacy effect)
• The opposite coul happen: the last option the interviewer read may be the first one that respondent think about (recency effect)
29. GUIDELINES FOR WRITING GOOD
QUESTIONS (Sudman and Bradburn)
Non sensitive questions about behavior
The key problem with many questions about behavios is that respondents may forget some or all of
the relevant information, or that their answer may reflect inaccurate estimate
In order to reduce memory problems it is essential to play attention to the wording of the
question and to provide memory help
Attitude questions
Attitude questions are a very commen class of survey questions. The most frequent problems deals with the wording of
questions, the question order and the format of response scales
30. Non sensitive questions about behavior
Play attention to the wording
With closed questions, include all reasonable
possibilities as explicit response options
• Are you:
• Married Are you:
• Divorced • Married
• Widowed • Single
• Separated
• Never married
31. Non sensitive questions about behavior
Play attention to the wording
Make the question as specific as possible
(about who it covers, what time period, which behaviours…)
Over the last month, In a tipical
that is ….. how week, how
often do you read a often do you
newspaper in a read a
tipical week? newspaper?
32. Non sensitive questions about behavior
Play attention to the wording
Use words that virtually all respondents will understand
Have you ever
Have you ever
had a heart had a miocardial
attack? infarction?
33. Non sensitive questions about behavior
Provide memory help
Uses aided recall
(or ask separate questions about subcategories)
Please look cerefully at the
following list of volountary
To which volountary
organisations: which, if any, organisation do you belong
do you belong to? to?
A Religious organisations
B Cultural organisations
C Political groups
D Other
34. Attitude questions
Play attention to the wording
Clearly specify the attitude object of interest
Do you think the Do you think the
Government is spending too Government is spending too
litte, about the right amount, litte, about the right amount,
or too much on higher or too much on education?
education?
35. Attitude questions*
play attention to the wording
Measure the strength of the attitute
using a response scale, a separate item
or multiple items that can be combined into a scale
Do you agree or disagree with the following statement?
Government is spending too little on education
1 Agree strongly
2 Agree
3 Neither agree nor disagree
4 Disagree
5 Disagree strongly
*note that an attitude have generally a direction (agree or disagree)
and an intensity (strongly disagree….strongly agree)
36. Attitude questions
reduce impact of question order
When asking general and specific questions about a
topic, ask the general question first
(otherwise, the answer to the general question is likely to be affected by the
number and content of specific questions)
When asking questions about about multiple items,
start with the least popular
(the unpopular questions are likely to seem even less appealing
when they follow more popular questions)
37. When asking general and specific questions about
a topic, ask the general question first
Please tell me whether or not you think it should be
possible for a pregnant woman to obtain a legal
abortion if: the woman wants it for any reason?
1. Yes
2. No
3. Don't know
Please tell me whether or not you think it should be possible
for a pregnant woman to obtain a legal abortion if: there is
a strong chance of a serious defect in the baby?
1. Yes
2. No
3. Don't know
USA GSS
38. Attitude questions
play attention to the response scale
Use closed questions for measuring aptitudes;
(open answers are difficult to code)
Use five-to seven-point response scales
and label every scale point
(verbal label ensure that interpret the scale in the same way)
